Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  DanSandClaude Opus 4.8
|
416f1ffd6d | ||
|
DanSandClaude Opus 5.5
|
fd67a1a15a | ||
|
DanSandClaude Opus 5.5
|
1faa349c27 | ||
|
DanSandClaude Opus 5.5
|
7f64df008e | ||
|
DanSandClaude Opus 5.5
|
c32c85b7f0 | ||
|
DanSandClaude Opus 5.5
|
1541d045ee | ||
|
DanSandClaude Opus 5.5
|
23f3e26d2f | ||
|
DanSandClaude Opus 5.5
|
08afe5bb3b | ||
|
DanSandClaude Opus 5.5
|
375b995b38 | ||
|
DanSandClaude Opus 5.5
|
106fc1ceac |
@@ -3,7 +3,7 @@
|
||||
.SH NAME
|
||||
dragonx-cli \- manual page for dragonx-cli v1.3.0
|
||||
.SH DESCRIPTION
|
||||
DragonX RPC client version v1.3.0
|
||||
DragonX RPC client version v1.3.0\-af7d9e230
|
||||
.PP
|
||||
In order to ensure you are adequately protecting your privacy when using
|
||||
DragonX, please see <https://dragonx.is/security/>.
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
.SH NAME
|
||||
dragonx-tx \- manual page for dragonx-tx v1.3.0
|
||||
.SH DESCRIPTION
|
||||
hush\-tx utility version v1.3.0
|
||||
hush\-tx utility version v1.3.0\-af7d9e230
|
||||
.SS "Usage:"
|
||||
.TP
|
||||
hush\-tx [options] <hex\-tx> [commands]
|
||||
|
||||
+1
-1
@@ -3,7 +3,7 @@
|
||||
.SH NAME
|
||||
dragonxd \- manual page for dragonxd v1.3.0
|
||||
.SH DESCRIPTION
|
||||
DragonX Daemon version v1.3.0
|
||||
DragonX Daemon version v1.3.0\-af7d9e230
|
||||
.PP
|
||||
In order to ensure you are adequately protecting your privacy when using
|
||||
DragonX, please see <https://dragonx.is/security/>.
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
shim/libfdguard.so
|
||||
shim/fdguard_test
|
||||
*.o
|
||||
shim/t/
|
||||
@@ -1,55 +0,0 @@
|
||||
# fd-guard: catch stray close() of files a thread doesn't own
|
||||
|
||||
This is the harness that found the wolfSSL `CloseSocket` macro bug. With that bug, P2P disconnects closed their fd number a second time, after another thread had reused it, and BDB then wrote wallet pages into `blocks/*.dat`.
|
||||
|
||||
It has two parts:
|
||||
|
||||
- `shim/`: an `LD_PRELOAD` guard around `close`, `fclose`, `dup2` and `dup3`.
|
||||
- Before each close it resolves `/proc/self/fd/N`.
|
||||
- It logs a backtrace when a non-owner closes `wallet.dat`, `database/log.*` or `blocks/*.dat`, and for every close that fails with `EBADF` (the second half of a double close).
|
||||
- `FDGUARD_ABORT=1` makes the first hit abort.
|
||||
- `repro/`: isolated regtest runs that put a shimmed `dragonxd` under load:
|
||||
- **rpc:** impatient and RST clients, half-open connections, work-queue bursts and long-polls;
|
||||
- **p2p:** TLS peers churned by `disconnectnode`/ban/kill, plus misbehaving raw TCP/TLS clients;
|
||||
- **tor:** a fake control port;
|
||||
- **mining:** with `blocknotify` and autoshield.
|
||||
|
||||
Each run executes in its own unprivileged network namespace: loopback only, no route.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Linux.
|
||||
- `unshare -rn` (unprivileged user namespaces), `nsenter`, `ip`, python3, gcc, and `addr2line` for symbolizing.
|
||||
- A `dragonxd`/`dragonx-cli` built with `-g`. Adding `-rdynamic` makes backtraces readable without `addr2line`.
|
||||
- A regtest datadir with a grown wallet to copy from (`GOLDEN`). It is never modified.
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
qa/fd-guard/shim/build.sh # builds libfdguard.so
|
||||
qa/fd-guard/shim/run_tests.sh # self-test: double close, foreign close of a guarded file
|
||||
cd qa/fd-guard/repro
|
||||
export GOLDEN=/path/to/regtest/datadir BIN=/path/to/build/src # RUNS defaults to ~/fd-guard-runs
|
||||
LOADS=p2pmin,mine DISC_INT=5 GUARD=all ./go.sh p2pmin 300 # 1 peer, disconnectnode every 5 s
|
||||
LOADS=rpc,p2p,tor,mine WQ=64 RT=8 GUARD=all ./go.sh full 1200 # everything, production-like work queue
|
||||
python3 tools/tally.py "$RUNS/<name>/fdguard.log" # who closed what
|
||||
python3 ../shim/symbolize.py --summary "$RUNS/<name>/fdguard.log"
|
||||
python3 tools/scan_bdb.py "$RUNS/<name>"/target/regtest/blocks/*.dat # BDB pages inside block files
|
||||
```
|
||||
|
||||
Each run writes `run.txt` (full log plus summary), `fdguard.log` and the target's `debug.log` and `db.log` under `$RUNS/<name>/`.
|
||||
|
||||
## What to expect
|
||||
|
||||
Measured on a 2041-note regtest wallet, before and after the fix (`net: stop wolfSSL's CloseSocket macro ...`), under the same loads:
|
||||
|
||||
| Run | Binary | EBADF closes | Non-owner closes (wallet/log/blk) | libevent `Epoll` failures | `db.log` |
|
||||
|---|---|---|---|---|---|
|
||||
| `p2pmin`, 300 s | before | 51 (≈ 1 per disconnect) | 1 | 0 | empty |
|
||||
| | after | 0 | 0 | 0 | empty |
|
||||
| `full`, 1200 s, `WQ=64 RT=8` | before | 930 | 186 (37) | 111 | 632 KB: 7500 × `BDB0060 PANIC`, `write failed for page N` |
|
||||
| | after | 0 | 0 | 0 | empty |
|
||||
|
||||
- RPC-only and Tor-only loads give 0 reports, even before the fix: P2P disconnects are the only origin.
|
||||
- `scan_bdb.py` found no BDB pages in block files in any run. The landing of a page in a block file needs the reused number to be a writable block file at the moment BDB writes. Production did hit that (blk00011/blk00020), but it is rare.
|
||||
- `SSL_write ... closing connection` lines are expected under `tcpchurn`. They are how the load triggers disconnects, not a symptom.
|
||||
@@ -1,72 +0,0 @@
|
||||
# shellcheck shell=bash
|
||||
# Shared config + helpers for the fd-double-close REPRODUCTION harness.
|
||||
# Regtest only. Meant to run INSIDE an unprivileged network namespace (see ns.sh),
|
||||
# so every port is private to the namespace and nothing can reach a real network.
|
||||
set +e
|
||||
|
||||
REPRO=${REPRO:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)}
|
||||
SRCROOT=$(cd "$REPRO/../../.." && pwd)
|
||||
BIN=${BIN:-$SRCROOT/src} # dir with dragonxd + dragonx-cli (build with -g -rdynamic for backtraces)
|
||||
DRAGONXD=$BIN/dragonxd
|
||||
CLI=$BIN/dragonx-cli
|
||||
RUNS=${RUNS:-$HOME/fd-guard-runs} # run output (block files, logs): keep it out of the repo
|
||||
GOLDEN=${GOLDEN:?set GOLDEN to a regtest datadir with a grown wallet (it is only ever copied)}
|
||||
ASMAP=${ASMAP:-$SRCROOT/asmap.dat}
|
||||
SP=${SP:-$HOME/.zcash-params/sapling-spend.params}
|
||||
OP=${OP:-$HOME/.zcash-params/sapling-output.params}
|
||||
SHIM=${SHIM:-$REPRO/../shim/libfdguard.so}
|
||||
|
||||
RPCUSER=ab
|
||||
RPCPASS=abpass
|
||||
RPC_T=18901 ; P2P_T=18911
|
||||
RPC_H1=18902; P2P_H1=18912
|
||||
RPC_H2=18903; P2P_H2=18913
|
||||
TORPORT=18950
|
||||
|
||||
clin() { local dd=$1 rp=$2; shift 2; "$CLI" -regtest -datadir="$dd" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$rp "$@"; }
|
||||
|
||||
write_conf() { printf 'rpcuser=%s\nrpcpassword=%s\nrpcport=%s\n' "$RPCUSER" "$RPCPASS" "$2" > "$1/DRAGONX.conf"; }
|
||||
|
||||
# prep_golden <dd> : copy of the golden wallet datadir (2041 notes, txindex=1, height 286)
|
||||
prep_golden() {
|
||||
local dd=$1
|
||||
rm -rf "$dd"; cp -a "$GOLDEN" "$dd"
|
||||
cp -f "$ASMAP" "$dd/asmap.dat"; cp -f "$SP" "$dd/sapling-spend.params"; cp -f "$OP" "$dd/sapling-output.params"
|
||||
write_conf "$dd" "$RPC_T"
|
||||
: > "$dd/regtest/debug.log"; rm -f "$dd/regtest/db.log"
|
||||
}
|
||||
|
||||
# prep_fresh <dd> <rpcport> : empty regtest datadir (helpers)
|
||||
prep_fresh() {
|
||||
local dd=$1 rp=$2
|
||||
rm -rf "$dd"; mkdir -p "$dd"
|
||||
cp -f "$ASMAP" "$dd/asmap.dat"; cp -f "$SP" "$dd/sapling-spend.params"; cp -f "$OP" "$dd/sapling-output.params"
|
||||
write_conf "$dd" "$rp"
|
||||
}
|
||||
|
||||
start_helper() { # <dd> <p2p> <rpc> <logfile>
|
||||
local dd=$1 p2p=$2 rp=$3 lg=$4
|
||||
( cd "$dd" && exec nice -n 10 "$DRAGONXD" -regtest -datadir="$dd" -testnode=1 -checkpoints=0 \
|
||||
-listen=1 -bind=127.0.0.1 -port=$p2p -connect=127.0.0.1:1 -addnode=127.0.0.1:$P2P_T \
|
||||
-dnsseed=0 -dns=0 -asmap="$dd/asmap.dat" -autoshield=0 \
|
||||
-rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$rp -listenonion=0 -maxinboundfromip=100 \
|
||||
-debug=net -debug=tls -daemon ) >> "$lg" 2>&1
|
||||
}
|
||||
|
||||
waitrpc() { # <dd> <rpc> [secs]
|
||||
local dd=$1 rp=$2 n=${3:-300} i
|
||||
for i in $(seq 1 "$n"); do clin "$dd" "$rp" getblockcount >/dev/null 2>&1 && return 0; sleep 1; done
|
||||
return 1
|
||||
}
|
||||
|
||||
stop_daemon() { # <dd> <rpc> (helpers, started with -daemon)
|
||||
local dd=$1 rp=$2 pid i
|
||||
pid=$(cat "$dd/regtest/hushd.pid" 2>/dev/null)
|
||||
clin "$dd" "$rp" stop >/dev/null 2>&1
|
||||
for i in $(seq 1 90); do { [ -z "$pid" ] || ! kill -0 "$pid" 2>/dev/null; } && return 0; sleep 1; done
|
||||
[ -n "$pid" ] && kill -9 "$pid" 2>/dev/null
|
||||
}
|
||||
|
||||
MARKERS=('libevent: Epoll' 'Socket close failed' 'Bad file descriptor' 'Operation not permitted' 'Illegal seek'
|
||||
'SSL_write' 'Unable to seek' 'socket select error' 'write failed for page' 'PANIC' 'error scanning DB'
|
||||
'terminate' 'Assertion' 'EXCEPTION')
|
||||
@@ -1,14 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# go.sh <name> <seconds> -- create a private network namespace (loopback only, no route),
|
||||
# run run.sh inside it, then tear the namespace down. Env knobs pass through to run.sh.
|
||||
export REPRO=${REPRO:-$(cd "$(dirname "$0")" && pwd)}
|
||||
export RUNS=${RUNS:-$HOME/fd-guard-runs}
|
||||
OUT=$RUNS/$1; mkdir -p "$OUT"
|
||||
setsid unshare -rn sh -c "ip link set lo up && echo \$\$ > $OUT/ns.pid && exec sleep infinity" </dev/null >/dev/null 2>&1 &
|
||||
for _ in 1 2 3 4 5 6 7 8 9 10; do [ -s "$OUT/ns.pid" ] && break; sleep 0.3; done
|
||||
NSP=$(cat "$OUT/ns.pid")
|
||||
cp "$REPRO/run.sh" "$OUT/run.sh.snapshot" # run a frozen copy: editing run.sh cannot disturb a live run
|
||||
nsenter -t "$NSP" -U -n --preserve-credentials bash "$OUT/run.sh.snapshot" "$@" > "$OUT/go.txt" 2>&1
|
||||
rc=$?
|
||||
kill "$NSP" 2>/dev/null
|
||||
echo "go.sh: run.sh rc=$rc, namespace holder $NSP killed" >> "$OUT/go.txt"
|
||||
@@ -1,156 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# run.sh <name> <seconds> -- run INSIDE the network namespace (go.sh does that).
|
||||
# Env knobs:
|
||||
# LOADS=rpc,p2p,tor,mine which load generators run (default all four); p2pmin = 1 helper + disconnectnode every DISC_INT s
|
||||
# WQ=4 RT=2 -rpcworkqueue / -rpcthreads of the target
|
||||
# GUARD=all FDGUARD_GUARD (db,blk | all)
|
||||
# SHIMON=1 run the target under LD_PRELOAD=libfdguard.so
|
||||
# STRACE=0 1 = also wrap the target in strace (failed fd syscalls only)
|
||||
# TRACE=0 1 = FDGUARD_TRACE_ALL (one line per open/close; big)
|
||||
# MINE_INT=3 seconds between generate 1
|
||||
# HAMMER_ARGS=... override the hammer thread mix
|
||||
# STOP_ON_HIT=0 1 = end the run 60 s after the first NON-OWNER db/blk close
|
||||
# EXTRA=... extra target args
|
||||
source "$REPRO/common.sh"
|
||||
NAME=$1; DUR=${2:-600}
|
||||
OUT=$RUNS/$NAME
|
||||
LOADS=${LOADS:-rpc,p2p,tor,mine}; WQ=${WQ:-4}; RT=${RT:-2}; GUARD=${GUARD:-all}
|
||||
SHIMON=${SHIMON:-1}; STRACE=${STRACE:-0}; TRACE=${TRACE:-0}; MINE_INT=${MINE_INT:-3}
|
||||
STOP_ON_HIT=${STOP_ON_HIT:-0}
|
||||
HAMMER_ARGS=${HAMMER_ARGS:---normal 8 --impatient 8 --slowfire 2 --halfopen 3 --pipeline 3 --garbage 2 --slowread 2 --longpoll 4 --burst-size 16 --burst-every 2}
|
||||
has() { case ",$LOADS," in *",$1,"*) return 0;; esac; return 1; }
|
||||
|
||||
mkdir -p "$OUT/logs"; rm -f "$OUT/STOP"
|
||||
DD_T=$OUT/target; DD_H1=$OUT/helper1; DD_H2=$OUT/helper2
|
||||
exec > >(tee -a "$OUT/run.txt") 2>&1
|
||||
echo "=== repro run $NAME $(date -u +%FT%TZ) dur=${DUR}s loads=$LOADS wq=$WQ rt=$RT guard=$GUARD shim=$SHIMON strace=$STRACE"
|
||||
echo "netns check (must be ENETUNREACH=101): $(python3 -c 'import socket;print(socket.socket().connect_ex(("1.1.1.1",80)))')"
|
||||
echo "binary: $DRAGONXD build-id: $(readelf -n "$DRAGONXD" | awk '/Build ID/{print $3}')"
|
||||
echo "dragonxd already running in this namespace? $(pgrep -a -x dragonxd | wc -l) (other agents' nodes live in other namespaces)"
|
||||
|
||||
PIDS=()
|
||||
cleanup() {
|
||||
echo "--- cleanup $(date -u +%T)"
|
||||
touch "$OUT/STOP"
|
||||
for p in "${PIDS[@]}"; do kill "$p" 2>/dev/null; done
|
||||
stop_daemon "$DD_H2" "$RPC_H2"; stop_daemon "$DD_H1" "$RPC_H1"
|
||||
if [ -n "$TPID" ] && kill -0 "$TPID" 2>/dev/null; then
|
||||
timeout 20 "$CLI" -regtest -datadir="$DD_T" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T stop >/dev/null 2>&1
|
||||
for _ in $(seq 1 60); do kill -0 "$TPID" 2>/dev/null || break; sleep 1; done
|
||||
# abandoned longpoll getblocktemplate calls can hold every -rpcthreads worker, so RPC stop may never run:
|
||||
# fall back to SIGTERM (same clean StartShutdown path), SIGKILL only as a last resort
|
||||
kill -0 "$TPID" 2>/dev/null && { echo "RPC stop did not finish in 60 s -> SIGTERM"; kill -TERM "$TPID"; }
|
||||
for _ in $(seq 1 180); do kill -0 "$TPID" 2>/dev/null || break; sleep 1; done
|
||||
kill -0 "$TPID" 2>/dev/null && { echo "target did not stop -> SIGKILL"; kill -9 "$TPID"; }
|
||||
fi
|
||||
[ -n "$WPID" ] && wait "$WPID" 2>/dev/null; echo "target exit status: $?"
|
||||
for p in "${PIDS[@]}"; do kill -9 "$p" 2>/dev/null; done
|
||||
summarize
|
||||
}
|
||||
summarize() {
|
||||
local DL=$DD_T/regtest/debug.log
|
||||
echo "=== markers in target debug.log"
|
||||
for m in "${MARKERS[@]}"; do printf ' [%6d] %s\n' "$(grep -c -- "$m" "$DL" 2>/dev/null)" "$m"; done
|
||||
echo " db.log bytes: $(stat -c %s "$DD_T/regtest/db.log" 2>/dev/null || echo absent)"
|
||||
grep -q 'Shutdown: done' "$DL" && echo " clean shutdown: yes" || echo " clean shutdown: NO"
|
||||
if [ "$SHIMON" = 1 ]; then
|
||||
echo "=== fdguard reports"; grep -o '=== FDGUARD #[0-9]* [A-Z-]*' "$OUT/fdguard.log" | awk '{print $4}' | sort | uniq -c
|
||||
grep '^S ' "$OUT/fdguard.log" | tail -1
|
||||
python3 "$REPRO/../shim/symbolize.py" --summary "$OUT/fdguard.log" > "$OUT/fdguard.summary.txt" 2>&1
|
||||
head -60 "$OUT/fdguard.summary.txt"
|
||||
fi
|
||||
echo "=== BDB-page-in-blockfile detector"
|
||||
python3 "$REPRO/tools/scan_bdb.py" "$DD_T"/regtest/blocks/blk*.dat "$DD_T"/regtest/blocks/rev*.dat 2>&1 | tail -15
|
||||
echo "=== end $(date -u +%FT%TZ)"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
trap 'exit 130' INT TERM
|
||||
|
||||
echo "--- prep datadirs"
|
||||
prep_golden "$DD_T"
|
||||
prep_fresh "$DD_H1" "$RPC_H1"; prep_fresh "$DD_H2" "$RPC_H2"
|
||||
cat > "$OUT/blocknotify.sh" <<EOF
|
||||
#!/bin/sh
|
||||
# pool-like blocknotify: a couple of dragonx-cli calls per block
|
||||
$CLI -regtest -datadir=$DD_T -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T getblock "\$1" >/dev/null 2>&1
|
||||
$CLI -regtest -datadir=$DD_T -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T getblocktemplate >/dev/null 2>&1
|
||||
echo "\$(date +%s) \$1" >> $OUT/logs/blocknotify.log
|
||||
EOF
|
||||
chmod +x "$OUT/blocknotify.sh"
|
||||
|
||||
if has tor; then
|
||||
nice -n 10 python3 "$REPRO/tools/faketor.py" --port $TORPORT > "$OUT/logs/faketor.txt" 2>&1 & PIDS+=($!)
|
||||
TORARGS="-listenonion=1 -torcontrol=127.0.0.1:$TORPORT -debug=tor"
|
||||
else
|
||||
TORARGS="-listenonion=0"
|
||||
fi
|
||||
if has p2p || has p2pmin; then P2PARGS="-listen=1 -bind=127.0.0.1 -port=$P2P_T -allowlist=127.0.0.0/8 -maxinboundfromip=100"
|
||||
else P2PARGS="-listen=0 -port=$P2P_T"; fi
|
||||
|
||||
TARGS=(-regtest -datadir="$DD_T" -testnode=1 -checkpoints=0 $P2PARGS -connect=127.0.0.1:1 -dnsseed=0 -dns=0
|
||||
-asmap="$DD_T/asmap.dat" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T
|
||||
-txindex=1 -autoshield=1 -rpcworkqueue=$WQ -rpcthreads=$RT
|
||||
"-blocknotify=$OUT/blocknotify.sh %s" $TORARGS -debug=net -debug=tls -printtoconsole=0 -daemon=0 $EXTRA)
|
||||
echo "target args: ${TARGS[*]}" | sed "s/$RPCPASS/***/g"
|
||||
ENVV=()
|
||||
if [ "$SHIMON" = 1 ]; then
|
||||
ENVV=(FDGUARD_LOG="$OUT/fdguard.log" FDGUARD_GUARD="$GUARD" FDGUARD_ABORT=0 FDGUARD_STATS_SEC=60 FDGUARD_MAXREPORTS=2000 LD_PRELOAD="$SHIM")
|
||||
[ "$TRACE" = 1 ] && ENVV+=(FDGUARD_TRACE_ALL=1)
|
||||
fi
|
||||
WRAP=()
|
||||
[ "$STRACE" = 1 ] && WRAP=(strace -f -tt -qq -e trace=close,openat,accept4,socket,dup2,dup3 -e status=failed -o "$OUT/strace.txt")
|
||||
( cd "$DD_T" && exec env "${ENVV[@]}" nice -n 10 "${WRAP[@]}" "$DRAGONXD" "${TARGS[@]}" ) > "$OUT/logs/target_stdout.txt" 2>&1 &
|
||||
WPID=$!
|
||||
echo "--- target launched (wrapper pid $WPID); waiting for RPC"
|
||||
waitrpc "$DD_T" "$RPC_T" 600 || { echo "target never answered RPC"; exit 1; }
|
||||
TPID=$(cat "$DD_T/regtest/hushd.pid" 2>/dev/null); [ -z "$TPID" ] && TPID=$WPID
|
||||
echo "target up pid=$TPID height=$(clin "$DD_T" "$RPC_T" getblockcount) $(date -u +%T)"
|
||||
|
||||
T0=$(date +%s)
|
||||
if has p2p; then
|
||||
start_helper "$DD_H1" "$P2P_H1" "$RPC_H1" "$OUT/logs/helper1.txt"
|
||||
start_helper "$DD_H2" "$P2P_H2" "$RPC_H2" "$OUT/logs/helper2.txt"
|
||||
waitrpc "$DD_H1" "$RPC_H1" 120; waitrpc "$DD_H2" "$RPC_H2" 120
|
||||
nice -n 10 python3 "$REPRO/tools/tcpchurn.py" --port $P2P_T --duration "$DUR" --concurrency 8 > "$OUT/logs/tcpchurn.txt" 2>&1 & PIDS+=($!)
|
||||
nice -n 10 bash "$REPRO/tools/rpcchurn.sh" "$OUT" "$DUR" > "$OUT/logs/rpcchurn.txt" 2>&1 & PIDS+=($!)
|
||||
fi
|
||||
if has p2pmin; then
|
||||
# minimal P2P: ONE well-behaved helper peer, and an ordinary "disconnectnode" every DISC_INT s
|
||||
# (the helper's -addnode re-dials the target). No raw TCP/TLS churn, no bans, no kills.
|
||||
start_helper "$DD_H1" "$P2P_H1" "$RPC_H1" "$OUT/logs/helper1.txt"; waitrpc "$DD_H1" "$RPC_H1" 120
|
||||
( k=0; while [ ! -e "$OUT/STOP" ]; do sleep "${DISC_INT:-5}"; for a in $(clin "$DD_T" "$RPC_T" getpeerinfo 2>/dev/null | python3 -c 'import json,sys; [print(p["addr"]) for p in json.load(sys.stdin)]' 2>/dev/null); do
|
||||
clin "$DD_T" "$RPC_T" disconnectnode "$a" >/dev/null 2>&1 && { k=$((k+1)); echo "$(date -u +%T.%N) disconnectnode $a" >> "$OUT/logs/disconnects.log"; }; done
|
||||
sleep 1; clin "$DD_H1" "$RPC_H1" addnode "127.0.0.1:$P2P_T" onetry >/dev/null 2>&1 # helper re-dials now (its -addnode retry is ~2 min)
|
||||
echo "$k" > "$OUT/logs/disconnects.txt"; done ) & PIDS+=($!)
|
||||
fi
|
||||
if has rpc; then
|
||||
MINEARG=""; has mine && MINEARG="--mine --mine-interval $MINE_INT"
|
||||
nice -n 10 python3 "$REPRO/tools/hammer.py" --port $RPC_T --user $RPCUSER --password $RPCPASS --duration "$DUR" \
|
||||
$HAMMER_ARGS $MINEARG --pid "$TPID" --logfile "$OUT/logs/hammer.log" > "$OUT/logs/hammer_stdout.txt" 2>&1 & PIDS+=($!)
|
||||
elif has mine; then
|
||||
( while [ ! -e "$OUT/STOP" ]; do clin "$DD_T" "$RPC_T" generate 1 >/dev/null 2>&1; sleep "$MINE_INT"; done ) & PIDS+=($!)
|
||||
fi
|
||||
|
||||
# ---- poll
|
||||
end=$(( T0 + DUR )); n=0; hit_at=0
|
||||
while [ "$(date +%s)" -lt "$end" ] && [ ! -e "$OUT/STOP" ]; do
|
||||
sleep 30; n=$((n+1))
|
||||
alive=no; kill -0 "$TPID" 2>/dev/null && alive=yes
|
||||
DL=$DD_T/regtest/debug.log
|
||||
h=$(clin "$DD_T" "$RPC_T" getblockcount 2>/dev/null); c=$(clin "$DD_T" "$RPC_T" getconnectioncount 2>/dev/null)
|
||||
rep=$(grep -c '^=== FDGUARD' "$OUT/fdguard.log" 2>/dev/null); noc=$(grep -c 'NON-OWNER-CLOSE' "$OUT/fdguard.log" 2>/dev/null)
|
||||
ebf=$(grep -c 'EBADF-CLOSE' "$OUT/fdguard.log" 2>/dev/null); epl=$(grep -c '=== FDGUARD #[0-9]* EPOLL' "$OUT/fdguard.log" 2>/dev/null)
|
||||
dbnoc=$(grep -A3 'NON-OWNER-CLOSE' "$OUT/fdguard.log" 2>/dev/null | grep -c 'class=\(db\|blk\)')
|
||||
printf '[poll %3d %s +%4ds] alive=%s h=%s conns=%s | fdguard reports=%s nonowner=%s(db/blk %s) ebadf=%s epoll=%s | Epoll=%s SockCloseFail=%s EBADF=%s EPERM=%s SSL_write=%s seek=%s sel=%s | db.log=%s\n' \
|
||||
$n "$(date -u +%T)" $(( $(date +%s) - T0 )) $alive "$h" "$c" "$rep" "$noc" "$dbnoc" "$ebf" "$epl" \
|
||||
"$(grep -c 'libevent: Epoll' "$DL")" "$(grep -c 'Socket close failed' "$DL")" "$(grep -c 'Bad file descriptor' "$DL")" \
|
||||
"$(grep -c 'Operation not permitted' "$DL")" "$(grep -c 'SSL_write' "$DL")" "$(grep -c 'Unable to seek' "$DL")" \
|
||||
"$(grep -c 'socket select error' "$DL")" "$(stat -c %s "$DD_T/regtest/db.log" 2>/dev/null)" | tee -a "$OUT/poll.txt"
|
||||
[ "$alive" = no ] && { echo "!!! target died"; break; }
|
||||
if [ "$STOP_ON_HIT" = 1 ] && [ "${dbnoc:-0}" -gt 0 ]; then
|
||||
[ "$hit_at" = 0 ] && hit_at=$(date +%s)
|
||||
[ $(( $(date +%s) - hit_at )) -ge 60 ] && { echo "--- stop-on-hit"; break; }
|
||||
fi
|
||||
done
|
||||
cat "$OUT/logs/tcpchurn.txt" "$OUT/logs/rpcchurn.txt" 2>/dev/null
|
||||
grep -E 'TOTAL|totals' -A12 "$OUT/logs/hammer.log" 2>/dev/null | tail -14
|
||||
@@ -1,98 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Fake Tor control port for the fd-bug harness.
|
||||
|
||||
dragonxd's torcontrol.cpp opens a libevent bufferevent to -torcontrol=<host:port>,
|
||||
speaks the control protocol (PROTOCOLINFO / AUTHCHALLENGE / AUTHENTICATE / ADD_ONION),
|
||||
and on EOF/ERROR reconnects with exponential backoff. We give it a server on a
|
||||
loopback port that behaves badly in several ways so the reconnect/teardown loop
|
||||
(bufferevent_free with BEV_OPT_CLOSE_ON_FREE, the event_base, the reconnect timer)
|
||||
runs constantly. Each accepted connection picks a behaviour at random:
|
||||
|
||||
close : accept, then immediately close (EOF right away)
|
||||
rst : accept, set SO_LINGER 0, close -> sends RST
|
||||
stall : accept, read nothing, hold the socket open then drop it
|
||||
garbage : send non-control-protocol bytes then close
|
||||
halfok : answer PROTOCOLINFO plausibly, then drop before AUTHENTICATE completes
|
||||
|
||||
Loopback only. Run with --port. Ctrl-C / SIGTERM stops it.
|
||||
"""
|
||||
import argparse, random, socket, struct, threading, time, sys, signal
|
||||
|
||||
STOP = threading.Event()
|
||||
|
||||
def handle(conn, mode):
|
||||
try:
|
||||
conn.settimeout(5.0)
|
||||
if mode == "close":
|
||||
conn.close(); return
|
||||
if mode == "rst":
|
||||
conn.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER,
|
||||
struct.pack("ii", 1, 0))
|
||||
conn.close(); return
|
||||
if mode == "stall":
|
||||
# read whatever the controller sends but never reply; hold 2-4s
|
||||
try:
|
||||
conn.recv(4096)
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(random.uniform(2.0, 4.0))
|
||||
conn.close(); return
|
||||
if mode == "garbage":
|
||||
try:
|
||||
conn.recv(4096)
|
||||
except Exception:
|
||||
pass
|
||||
conn.sendall(b"\x00\xff\x01garbage not a tor reply\r\n" * 4)
|
||||
conn.close(); return
|
||||
if mode == "halfok":
|
||||
# read the PROTOCOLINFO line, give a plausible 250 reply advertising
|
||||
# NULL auth, then drop the connection before AUTHENTICATE finishes.
|
||||
try:
|
||||
conn.recv(4096)
|
||||
except Exception:
|
||||
pass
|
||||
conn.sendall(b"250-PROTOCOLINFO 1\r\n"
|
||||
b"250-AUTH METHODS=NULL\r\n"
|
||||
b'250-VERSION Tor="0.4.7.13"\r\n'
|
||||
b"250 OK\r\n")
|
||||
time.sleep(random.uniform(0.2, 1.0))
|
||||
conn.close(); return
|
||||
except Exception:
|
||||
try: conn.close()
|
||||
except Exception: pass
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--port", type=int, required=True)
|
||||
ap.add_argument("--host", default="127.0.0.1")
|
||||
args = ap.parse_args()
|
||||
|
||||
modes = ["close", "rst", "stall", "garbage", "halfok"]
|
||||
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
srv.bind((args.host, args.port))
|
||||
srv.listen(16)
|
||||
srv.settimeout(1.0)
|
||||
print(f"faketor: listening on {args.host}:{args.port}", flush=True)
|
||||
|
||||
def onsig(*_): STOP.set()
|
||||
signal.signal(signal.SIGTERM, onsig)
|
||||
signal.signal(signal.SIGINT, onsig)
|
||||
|
||||
n = 0
|
||||
while not STOP.is_set():
|
||||
try:
|
||||
conn, _ = srv.accept()
|
||||
except socket.timeout:
|
||||
continue
|
||||
except Exception:
|
||||
break
|
||||
mode = modes[n % len(modes)]
|
||||
n += 1
|
||||
threading.Thread(target=handle, args=(conn, mode), daemon=True).start()
|
||||
srv.close()
|
||||
print(f"faketor: stopped after {n} connections", flush=True)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,796 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# hammer.py -- JSON-RPC robustness/load harness for a LOCAL regtest dragonxd.
|
||||
#
|
||||
# Scope: connects only to the --host/--port you pass (meant to be 127.0.0.1 and a
|
||||
# regtest RPC port). Its job is to reproduce a known file-descriptor bug in the
|
||||
# libevent-2.1.8 HTTP RPC server (src/httpserver.cpp) by exercising the same
|
||||
# request patterns the production pool box sees: many concurrent clients, clients
|
||||
# that disconnect mid-request/mid-response (the explorer indexer's 120 s client
|
||||
# timeout), work-queue overflow, keep-alive pipelining, and malformed input.
|
||||
# stdlib only.
|
||||
#
|
||||
# Correctness oracle: every request carries a unique JSON-RPC id; if a reply comes
|
||||
# back with a different id on the same socket, that is cross-talk (the server wrote
|
||||
# one client's reply onto another client's fd) and is logged + counted.
|
||||
#
|
||||
# See README.txt for the exact invocation.
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import collections
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import socket
|
||||
import struct
|
||||
import threading
|
||||
import time
|
||||
|
||||
# ------------------------------------------------------------------ config/state
|
||||
|
||||
class C:
|
||||
host = "127.0.0.1"
|
||||
port = 19461
|
||||
auth = ""
|
||||
ctimeout = 15.0
|
||||
|
||||
CFG = C()
|
||||
STOP = threading.Event()
|
||||
SHARED = {"hashes": [], "lpid": None, "zaddrs": [], "rawblock": None}
|
||||
SHARED_LOCK = threading.Lock()
|
||||
_ID = [random.SystemRandom().randrange(1 << 40)]
|
||||
_ID_LOCK = threading.Lock()
|
||||
LOGLOCK = threading.Lock()
|
||||
LOGF = [None]
|
||||
|
||||
def next_id():
|
||||
with _ID_LOCK:
|
||||
_ID[0] += 1
|
||||
return _ID[0]
|
||||
|
||||
def log(msg):
|
||||
line = "%s %s" % (time.strftime("%H:%M:%S"), msg)
|
||||
with LOGLOCK:
|
||||
print(line, flush=True)
|
||||
if LOGF[0]:
|
||||
LOGF[0].write(line + "\n"); LOGF[0].flush()
|
||||
|
||||
# ------------------------------------------------------------------ stats
|
||||
|
||||
KEYS = ["sent", "ok", "rpcerr", "wq500", "http401", "http4xx", "httpother",
|
||||
"early_rst", "early_fin", "srvreset", "srveof", "ctimeout", "connfail",
|
||||
"idmismatch", "malformed", "mined", "half_open", "half_srvclose"]
|
||||
|
||||
class Stats:
|
||||
def __init__(self):
|
||||
self.lk = threading.Lock()
|
||||
self.total = collections.Counter()
|
||||
self.win = collections.Counter()
|
||||
self.anom = []
|
||||
def inc(self, k, n=1):
|
||||
with self.lk:
|
||||
self.total[k] += n; self.win[k] += n
|
||||
def anomaly(self, kind, detail):
|
||||
with self.lk:
|
||||
if len(self.anom) < 300:
|
||||
self.anom.append((time.time(), kind, detail))
|
||||
log("ANOMALY %s %s" % (kind, detail))
|
||||
def swap(self):
|
||||
with self.lk:
|
||||
w = self.win; self.win = collections.Counter(); return w
|
||||
|
||||
ST = Stats()
|
||||
|
||||
# ------------------------------------------------------------------ socket helpers
|
||||
|
||||
def auth_header(u, p):
|
||||
return "Basic " + base64.b64encode(("%s:%s" % (u, p)).encode()).decode()
|
||||
|
||||
def mkconn(timeout=None, rcvbuf=None):
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
if rcvbuf:
|
||||
s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, rcvbuf)
|
||||
s.settimeout(timeout if timeout is not None else CFG.ctimeout)
|
||||
s.connect((CFG.host, CFG.port))
|
||||
return s
|
||||
|
||||
def end_rst(s):
|
||||
# linger 0 -> TCP RST on close (models a client that dies / resets abruptly)
|
||||
try:
|
||||
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
s.close()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def end_fin(s):
|
||||
try:
|
||||
s.shutdown(socket.SHUT_RDWR)
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
s.close()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def end_early(s, how=None):
|
||||
how = how or random.choice(("rst", "fin"))
|
||||
if how == "rst":
|
||||
end_rst(s); ST.inc("early_rst")
|
||||
else:
|
||||
end_fin(s); ST.inc("early_fin")
|
||||
|
||||
def req_bytes(method, params, rid, keepalive=True, auth=None, uri="/",
|
||||
httpmethod="POST", body=None, clen=None):
|
||||
if body is None:
|
||||
body = json.dumps({"jsonrpc": "1.0", "id": rid,
|
||||
"method": method, "params": params}).encode()
|
||||
head = [
|
||||
"%s %s HTTP/1.1" % (httpmethod, uri),
|
||||
"Host: %s:%d" % (CFG.host, CFG.port),
|
||||
"Authorization: %s" % (auth if auth is not None else CFG.auth),
|
||||
"Content-Type: application/json",
|
||||
"Content-Length: %d" % (clen if clen is not None else len(body)),
|
||||
"Connection: %s" % ("keep-alive" if keepalive else "close"),
|
||||
"", "",
|
||||
]
|
||||
return "\r\n".join(head).encode() + body
|
||||
|
||||
class Reader:
|
||||
"""Incremental HTTP/1.1 reply reader over one socket."""
|
||||
def __init__(self, s):
|
||||
self.s = s; self.buf = b""
|
||||
def _fill(self):
|
||||
d = self.s.recv(65536)
|
||||
if not d:
|
||||
raise EOFError()
|
||||
self.buf += d
|
||||
def read(self):
|
||||
while b"\r\n\r\n" not in self.buf:
|
||||
self._fill()
|
||||
if len(self.buf) > (1 << 20):
|
||||
raise ValueError("oversize header")
|
||||
head, self.buf = self.buf.split(b"\r\n\r\n", 1)
|
||||
lines = head.split(b"\r\n")
|
||||
p = lines[0].split(b" ", 2)
|
||||
if len(p) < 2 or not p[0].startswith(b"HTTP/1."):
|
||||
raise ValueError("bad status line %r" % lines[0][:80])
|
||||
status = int(p[1])
|
||||
hdr = {}
|
||||
for ln in lines[1:]:
|
||||
if b":" in ln:
|
||||
k, v = ln.split(b":", 1)
|
||||
hdr[k.strip().lower()] = v.strip()
|
||||
if hdr.get(b"transfer-encoding", b"").lower() == b"chunked":
|
||||
body = b""
|
||||
while True:
|
||||
while b"\r\n" not in self.buf:
|
||||
self._fill()
|
||||
ln, self.buf = self.buf.split(b"\r\n", 1)
|
||||
n = int(ln.split(b";")[0], 16)
|
||||
while len(self.buf) < n + 2:
|
||||
self._fill()
|
||||
body += self.buf[:n]; self.buf = self.buf[n + 2:]
|
||||
if n == 0:
|
||||
break
|
||||
elif b"content-length" in hdr:
|
||||
n = int(hdr[b"content-length"])
|
||||
while len(self.buf) < n:
|
||||
self._fill()
|
||||
body, self.buf = self.buf[:n], self.buf[n:]
|
||||
else:
|
||||
try:
|
||||
while True:
|
||||
self._fill()
|
||||
except EOFError:
|
||||
pass
|
||||
body, self.buf = self.buf, b""
|
||||
return status, hdr, body
|
||||
|
||||
def account(status, body, rid, method, expect401=False):
|
||||
if status == 401:
|
||||
ST.inc("http401")
|
||||
if not expect401:
|
||||
ST.anomaly("unexpected401", (method,))
|
||||
return None
|
||||
if status == 500 and b"Work queue depth exceeded" in body:
|
||||
ST.inc("wq500"); return None
|
||||
j = None
|
||||
if body[:1] in (b"{", b"["):
|
||||
try:
|
||||
j = json.loads(body)
|
||||
except ValueError:
|
||||
j = None
|
||||
if j is None:
|
||||
if 400 <= status < 500:
|
||||
ST.inc("http4xx")
|
||||
else:
|
||||
ST.inc("httpother")
|
||||
if status == 200:
|
||||
ST.anomaly("non-json-200", (method, body[:120].decode("latin1")))
|
||||
return None
|
||||
if isinstance(j, dict) and rid is not None and j.get("id") != rid:
|
||||
ST.inc("idmismatch")
|
||||
ST.anomaly("idmismatch", {"method": method, "sent": rid, "got": j.get("id"),
|
||||
"body": body[:200].decode("latin1")})
|
||||
if isinstance(j, dict) and j.get("error"):
|
||||
ST.inc("rpcerr")
|
||||
else:
|
||||
ST.inc("ok")
|
||||
return j
|
||||
|
||||
def call(s, rd, method, params, keepalive=True, expect401=False, auth=None):
|
||||
rid = next_id()
|
||||
try:
|
||||
s.sendall(req_bytes(method, params, rid, keepalive=keepalive, auth=auth))
|
||||
ST.inc("sent")
|
||||
status, hdr, body = rd.read()
|
||||
except socket.timeout:
|
||||
ST.inc("ctimeout"); return None, False
|
||||
except (ConnectionResetError, BrokenPipeError):
|
||||
ST.inc("srvreset"); return None, False
|
||||
except EOFError:
|
||||
ST.inc("srveof"); return None, False
|
||||
except ValueError as e:
|
||||
ST.inc("malformed"); ST.anomaly("malformed", (method, str(e))); return None, False
|
||||
except OSError:
|
||||
ST.inc("srvreset"); return None, False
|
||||
j = account(status, body, rid, method, expect401)
|
||||
alive = keepalive and hdr.get(b"connection", b"").lower() != b"close"
|
||||
return j, alive
|
||||
|
||||
# ------------------------------------------------------------------ menus
|
||||
|
||||
def a_hash():
|
||||
with SHARED_LOCK:
|
||||
return random.choice(SHARED["hashes"]) if SHARED["hashes"] else None
|
||||
|
||||
def a_zaddr():
|
||||
with SHARED_LOCK:
|
||||
return random.choice(SHARED["zaddrs"]) if SHARED["zaddrs"] else None
|
||||
|
||||
def normal_menu():
|
||||
# Mostly fast calls (sub-ms) so a well-behaved client interleaves with mining.
|
||||
# Heavy wallet calls (z_gettotalbalance ~2.3s on the golden wallet) appear but
|
||||
# are rare, so they don't permanently peg the small worker pool on their own.
|
||||
fast = [("getinfo", []), ("getblockcount", []), ("getbestblockhash", []),
|
||||
("getrawmempool", []), ("getmininginfo", []), ("getdifficulty", []),
|
||||
("getblockchaininfo", []), ("getpeerinfo", []),
|
||||
("getblocktemplate", []), ("getnetworkinfo", [])]
|
||||
h = a_hash()
|
||||
if h:
|
||||
fast.append(("getblock", [h, 2]))
|
||||
fast.append(("getblockheader", [h]))
|
||||
# "slow" here is only mildly-larger fast calls; the genuinely heavy wallet calls
|
||||
# live in the paced slow-fire class so a normal dashboard poller never pegs the
|
||||
# worker pool.
|
||||
slow = [("coinsupply", []), ("listtransactions", ["*", 10]),
|
||||
("listunspent", [0]), ("getblock", [a_hash() or "", 2])]
|
||||
return random.choice(slow) if random.random() < 0.1 else random.choice(fast)
|
||||
|
||||
def medium_menu():
|
||||
# Fast to compute (~4 ms on this node) but a non-trivial reply to write, so they
|
||||
# are good targets for a mid-response disconnect. The fd-reuse race fires on the
|
||||
# disconnect, not on call duration, so firing these at a HIGH rate drives the
|
||||
# race hard while the worker frees in milliseconds (mining keeps its slots).
|
||||
m = [("listtransactions", ["*", 10000]), ("getrawmempool", []),
|
||||
("getrawmempool", [True]), ("getblockchaininfo", []),
|
||||
("getblocktemplate", []), ("getchaintips", []), ("getinfo", []),
|
||||
("coinsupply", []), ("listunspent", [0])]
|
||||
h = a_hash()
|
||||
if h:
|
||||
m.append(("getblock", [h, 2]))
|
||||
m.append(("getblock", [h, 1]))
|
||||
return random.choice(m)
|
||||
|
||||
def heavy_menu():
|
||||
# Genuinely slow calls (z_gettotalbalance ~1.7 s, z_listunspent ~5.7 s on the
|
||||
# golden wallet). Fired only by the small, PACED slow-fire class so a worker
|
||||
# keeps running a multi-second call long after the client disconnected (the
|
||||
# production indexer/coinsupply scenario that widens the fd-reuse window),
|
||||
# without the fire-and-disconnect backlog pinning the whole pool.
|
||||
m = [("z_gettotalbalance", [0]), ("z_listunspent", [0])]
|
||||
z = a_zaddr()
|
||||
if z:
|
||||
m.append(("z_listreceivedbyaddress", [z, 0]))
|
||||
return random.choice(m)
|
||||
|
||||
# ------------------------------------------------------------------ clients
|
||||
|
||||
def client_normal(cid):
|
||||
while not STOP.is_set():
|
||||
keepalive = random.random() < 0.7
|
||||
try:
|
||||
s = mkconn()
|
||||
except OSError:
|
||||
ST.inc("connfail"); time.sleep(0.05); continue
|
||||
rd = Reader(s)
|
||||
n = random.randint(1, 8) if keepalive else 1
|
||||
try:
|
||||
for _ in range(n):
|
||||
if STOP.is_set():
|
||||
break
|
||||
method, params = normal_menu()
|
||||
j, alive = call(s, rd, method, params, keepalive=keepalive)
|
||||
if not alive:
|
||||
break
|
||||
finally:
|
||||
(end_fin if random.random() < 0.8 else end_rst)(s)
|
||||
time.sleep(random.uniform(0.01, 0.15))
|
||||
|
||||
def client_impatient(cid):
|
||||
# High-rate fast-disconnect churn: send a (medium) request, then disconnect
|
||||
# after 1-500 ms at a random phase. This is the main driver of the fd-reuse
|
||||
# race -- a disconnect concurrent with the server writing the reply / re-arming
|
||||
# the socket -- and the worker frees in ms so mining is unaffected.
|
||||
while not STOP.is_set():
|
||||
method, params = medium_menu()
|
||||
rid = next_id()
|
||||
body = json.dumps({"jsonrpc": "1.0", "id": rid, "method": method,
|
||||
"params": params}).encode()
|
||||
phase = random.choice(("mid_request", "after_send", "mid_response"))
|
||||
try:
|
||||
s = mkconn(timeout=2.0)
|
||||
except OSError:
|
||||
ST.inc("connfail"); continue
|
||||
try:
|
||||
if phase == "mid_request":
|
||||
full = req_bytes(method, params, rid, keepalive=False, body=body)
|
||||
cut = random.randint(len(full) - len(body), len(full) - 1)
|
||||
s.sendall(full[:cut])
|
||||
time.sleep(random.uniform(0.001, 0.5))
|
||||
end_early(s); continue
|
||||
s.sendall(req_bytes(method, params, rid, keepalive=False, body=body))
|
||||
ST.inc("sent")
|
||||
time.sleep(random.uniform(0.001, 0.5))
|
||||
if phase == "mid_response":
|
||||
try:
|
||||
s.recv(random.randint(0, 64)) # read a few bytes, then bail
|
||||
except OSError:
|
||||
pass
|
||||
end_early(s)
|
||||
except (OSError, socket.timeout):
|
||||
ST.inc("srvreset")
|
||||
end_rst(s)
|
||||
time.sleep(random.uniform(0.05, 0.4))
|
||||
|
||||
def client_slowfire(cid, pace):
|
||||
# Paced indexer/coinsupply client: fire ONE genuinely-slow request, disconnect
|
||||
# after 1-500 ms (so the worker runs the multi-second call with no client left
|
||||
# == the production fd-reuse window), then wait `pace` s before the next so a
|
||||
# fire-and-disconnect backlog cannot pin the whole worker pool. Phase: cut the
|
||||
# request mid-body, or send it whole and abort while the worker is busy, or read
|
||||
# a few bytes of the reply then abort.
|
||||
while not STOP.is_set():
|
||||
method, params = heavy_menu()
|
||||
rid = next_id()
|
||||
body = json.dumps({"jsonrpc": "1.0", "id": rid, "method": method,
|
||||
"params": params}).encode()
|
||||
phase = random.choice(("mid_request", "after_send", "mid_response"))
|
||||
try:
|
||||
s = mkconn(timeout=2.0)
|
||||
except OSError:
|
||||
ST.inc("connfail"); time.sleep(pace); continue
|
||||
try:
|
||||
if phase == "mid_request":
|
||||
full = req_bytes(method, params, rid, keepalive=False, body=body)
|
||||
cut = random.randint(len(full) - len(body), len(full) - 1)
|
||||
s.sendall(full[:cut])
|
||||
time.sleep(random.uniform(0.001, 0.5)); end_early(s)
|
||||
else:
|
||||
s.sendall(req_bytes(method, params, rid, keepalive=False, body=body))
|
||||
ST.inc("sent")
|
||||
time.sleep(random.uniform(0.001, 0.5))
|
||||
if phase == "mid_response":
|
||||
try:
|
||||
s.recv(random.randint(0, 64))
|
||||
except OSError:
|
||||
pass
|
||||
end_early(s)
|
||||
except (OSError, socket.timeout):
|
||||
ST.inc("srvreset"); end_rst(s)
|
||||
time.sleep(random.uniform(pace * 0.5, pace * 1.5))
|
||||
|
||||
def client_half_open(cid):
|
||||
# connect, send nothing / partial headers, trickle, then vanish
|
||||
while not STOP.is_set():
|
||||
try:
|
||||
s = mkconn(timeout=1.0)
|
||||
except OSError:
|
||||
ST.inc("connfail"); time.sleep(0.1); continue
|
||||
ST.inc("half_open")
|
||||
style = random.choice(("silent", "partial_headers", "full_headers_no_body"))
|
||||
try:
|
||||
if style == "partial_headers":
|
||||
s.sendall(b"POST / HTTP/1.1\r\nHost: x\r\n")
|
||||
elif style == "full_headers_no_body":
|
||||
s.sendall(b"POST / HTTP/1.1\r\nHost: x\r\nAuthorization: " +
|
||||
CFG.auth.encode() + b"\r\nContent-Length: 1000\r\n\r\n")
|
||||
deadline = time.monotonic() + random.uniform(0.1, 20.0)
|
||||
while time.monotonic() < deadline and not STOP.is_set():
|
||||
time.sleep(random.uniform(0.2, 1.5))
|
||||
try:
|
||||
s.sendall(b" ") # one byte, keep it half-alive
|
||||
except OSError:
|
||||
ST.inc("half_srvclose"); break
|
||||
try:
|
||||
s.settimeout(0.01)
|
||||
if s.recv(1) == b"":
|
||||
ST.inc("half_srvclose"); break
|
||||
except socket.timeout:
|
||||
pass
|
||||
except OSError:
|
||||
ST.inc("half_srvclose"); break
|
||||
finally:
|
||||
end_early(s)
|
||||
|
||||
def client_burst(cid, size, every):
|
||||
while not STOP.is_set():
|
||||
batch = []
|
||||
for _ in range(size):
|
||||
try:
|
||||
s = mkconn(timeout=3.0)
|
||||
except OSError:
|
||||
ST.inc("connfail"); continue
|
||||
batch.append(s)
|
||||
for s in batch:
|
||||
try:
|
||||
method, params = medium_menu()
|
||||
s.sendall(req_bytes(method, params, next_id(), keepalive=False))
|
||||
ST.inc("sent")
|
||||
except OSError:
|
||||
pass
|
||||
# most close immediately (before the 500/reply is written); a few read it
|
||||
keep = random.sample(batch, max(0, len(batch) // 4)) if batch else []
|
||||
for s in batch:
|
||||
if s in keep:
|
||||
continue
|
||||
end_early(s)
|
||||
for s in keep:
|
||||
try:
|
||||
rd = Reader(s); st, hd, bd = rd.read(); account(st, bd, None, "burst")
|
||||
except Exception:
|
||||
ST.inc("srvreset")
|
||||
finally:
|
||||
end_fin(s)
|
||||
for _ in range(int(every * 10)):
|
||||
if STOP.is_set():
|
||||
break
|
||||
time.sleep(0.1)
|
||||
|
||||
def client_pipeline(cid):
|
||||
while not STOP.is_set():
|
||||
try:
|
||||
s = mkconn()
|
||||
except OSError:
|
||||
ST.inc("connfail"); time.sleep(0.05); continue
|
||||
n = random.randint(2, 8)
|
||||
ids = []
|
||||
blob = b""
|
||||
for _ in range(n):
|
||||
rid = next_id(); ids.append(rid)
|
||||
method, params = normal_menu()
|
||||
blob += req_bytes(method, params, rid, keepalive=True)
|
||||
try:
|
||||
s.sendall(blob); ST.inc("sent", n)
|
||||
rd = Reader(s)
|
||||
toread = random.randint(0, n) # sometimes leave replies unread, then close
|
||||
for k in range(toread):
|
||||
st, hd, bd = rd.read()
|
||||
account(st, bd, ids[k], "pipeline")
|
||||
except (OSError, EOFError, ValueError):
|
||||
ST.inc("srvreset")
|
||||
finally:
|
||||
end_early(s)
|
||||
time.sleep(random.uniform(0, 0.03))
|
||||
|
||||
def client_garbage(cid):
|
||||
probes = [
|
||||
("submitblock", lambda: ["ab" * random.randint(1, 4000)]),
|
||||
("submitblock", lambda: [os.urandom(random.randint(1, 200)).hex()]),
|
||||
("submitblock", lambda: ["zzzz"]),
|
||||
("submitblock", lambda: [""]),
|
||||
]
|
||||
while not STOP.is_set():
|
||||
pick = random.random()
|
||||
try:
|
||||
s = mkconn(timeout=5.0)
|
||||
except OSError:
|
||||
ST.inc("connfail"); continue
|
||||
try:
|
||||
if pick < 0.5:
|
||||
method, mk = random.choice(probes)
|
||||
j, _ = call(s, Reader(s), method, mk(), keepalive=False)
|
||||
elif pick < 0.65:
|
||||
# malformed JSON body
|
||||
s.sendall(req_bytes("x", [], next_id(), keepalive=False,
|
||||
body=b'{"jsonrpc":"1.0","method":'))
|
||||
ST.inc("sent")
|
||||
try:
|
||||
account(*((lambda r: (r[0], r[2]))(Reader(s).read())), rid=None, method="badjson")
|
||||
except Exception:
|
||||
ST.inc("srvreset")
|
||||
elif pick < 0.8:
|
||||
# unknown method / wrong URI
|
||||
s.sendall(req_bytes("getinfo", [], next_id(), keepalive=False, uri="/nope"))
|
||||
ST.inc("sent")
|
||||
try:
|
||||
st, hd, bd = Reader(s).read(); account(st, bd, None, "wronguri")
|
||||
except Exception:
|
||||
ST.inc("srvreset")
|
||||
else:
|
||||
# Content-Length larger than the body we send, then bail
|
||||
body = b'{"id":1}'
|
||||
s.sendall(req_bytes("getinfo", [], 1, keepalive=False, body=body, clen=len(body) + 5000))
|
||||
ST.inc("sent")
|
||||
time.sleep(random.uniform(0.001, 0.3))
|
||||
end_early(s)
|
||||
continue
|
||||
except (OSError, socket.timeout):
|
||||
ST.inc("srvreset")
|
||||
finally:
|
||||
end_early(s)
|
||||
time.sleep(random.uniform(0.02, 0.2))
|
||||
|
||||
def client_slowread(cid):
|
||||
# ask for a big reply with a tiny recv buffer and do not read it, stalling the
|
||||
# server write side, then close abruptly
|
||||
while not STOP.is_set():
|
||||
try:
|
||||
s = mkconn(timeout=3.0, rcvbuf=512)
|
||||
except OSError:
|
||||
ST.inc("connfail"); continue
|
||||
try:
|
||||
s.sendall(req_bytes("listtransactions", ["*", 10000], next_id(), keepalive=False))
|
||||
ST.inc("sent")
|
||||
time.sleep(random.uniform(0.05, 1.0))
|
||||
except OSError:
|
||||
ST.inc("srvreset")
|
||||
finally:
|
||||
end_early(s)
|
||||
time.sleep(random.uniform(0, 0.05))
|
||||
|
||||
def _stale_lpid():
|
||||
# A longpollid whose hash != the current tip makes getblocktemplate return at
|
||||
# once (the server's wait loop condition is immediately false), so it exercises
|
||||
# the long-poll request path (lpval parsing, cs_main leave/enter) and the
|
||||
# mid-response disconnect WITHOUT parking a worker. On a tiny -rpcthreads pool a
|
||||
# genuinely parking long-poll + RPC-driven mining self-deadlocks (both workers
|
||||
# wait for a block that only a third worker could mine); that is a pool-size
|
||||
# artifact, not the fd bug, so the harness avoids it by default.
|
||||
with SHARED_LOCK:
|
||||
lpid = SHARED["lpid"]
|
||||
if lpid and len(lpid) > 64:
|
||||
h = lpid[:64]
|
||||
flip = "0" if h[0] != "0" else "1"
|
||||
return flip + h[1:] + lpid[64:]
|
||||
return ("0" * 64) + "0"
|
||||
|
||||
def client_longpoll(cid):
|
||||
while not STOP.is_set():
|
||||
try:
|
||||
s = mkconn(timeout=10.0)
|
||||
except OSError:
|
||||
ST.inc("connfail"); time.sleep(0.2); continue
|
||||
rd = Reader(s)
|
||||
params = [{"longpollid": _stale_lpid()}]
|
||||
rid = next_id()
|
||||
try:
|
||||
s.sendall(req_bytes("getblocktemplate", params, rid, keepalive=False))
|
||||
ST.inc("sent")
|
||||
if random.random() < 0.5:
|
||||
# impatient pool worker: abandon the request after 1-500 ms
|
||||
time.sleep(random.uniform(0.001, 0.5))
|
||||
end_early(s); continue
|
||||
st, hd, bd = rd.read()
|
||||
j = account(st, bd, rid, "getblocktemplate(lp)")
|
||||
if j and isinstance(j, dict) and j.get("result"):
|
||||
with SHARED_LOCK:
|
||||
SHARED["lpid"] = j["result"].get("longpollid")
|
||||
except (OSError, socket.timeout, EOFError, ValueError):
|
||||
ST.inc("srvreset")
|
||||
finally:
|
||||
end_fin(s)
|
||||
time.sleep(random.uniform(0.02, 0.2))
|
||||
|
||||
def miner(interval):
|
||||
# Mine a block every `interval` seconds. Retry through work-queue rejections so
|
||||
# blocks still land under heavy load (the production box mines steadily too).
|
||||
while not STOP.is_set():
|
||||
got = False
|
||||
deadline = time.monotonic() + interval + 30
|
||||
while not got and not STOP.is_set() and time.monotonic() < deadline:
|
||||
try:
|
||||
s = mkconn(timeout=60.0)
|
||||
j, _ = call(s, Reader(s), "generate", [1], keepalive=False)
|
||||
end_fin(s)
|
||||
if j and not j.get("error") and j.get("result"):
|
||||
ST.inc("mined"); got = True
|
||||
h = j["result"][0] if isinstance(j["result"], list) else None
|
||||
if h:
|
||||
with SHARED_LOCK:
|
||||
SHARED["hashes"].append(h)
|
||||
SHARED["hashes"] = SHARED["hashes"][-200:]
|
||||
else:
|
||||
time.sleep(0.2) # likely work-queue 500; back off and retry
|
||||
except Exception:
|
||||
time.sleep(0.2)
|
||||
for _ in range(int(interval * 10)):
|
||||
if STOP.is_set():
|
||||
break
|
||||
time.sleep(0.1)
|
||||
|
||||
# ------------------------------------------------------------------ bootstrap shared state
|
||||
|
||||
def bootstrap():
|
||||
try:
|
||||
s = mkconn(timeout=30.0); rd = Reader(s)
|
||||
j, _ = call(s, rd, "getbestblockhash", [], keepalive=True)
|
||||
if j and j.get("result"):
|
||||
with SHARED_LOCK:
|
||||
SHARED["hashes"].append(j["result"])
|
||||
j, _ = call(s, rd, "getblockcount", [], keepalive=True)
|
||||
cnt = j.get("result") if j else None
|
||||
if isinstance(cnt, int):
|
||||
for hh in range(max(0, cnt - 30), cnt + 1):
|
||||
jj, _ = call(s, rd, "getblockhash", [hh], keepalive=True)
|
||||
if jj and jj.get("result"):
|
||||
with SHARED_LOCK:
|
||||
SHARED["hashes"].append(jj["result"])
|
||||
j, _ = call(s, rd, "listaddresses", [], keepalive=True)
|
||||
j, _ = call(s, rd, "z_listaddresses", [], keepalive=True)
|
||||
if j and isinstance(j.get("result"), list):
|
||||
with SHARED_LOCK:
|
||||
SHARED["zaddrs"] = j["result"][:16]
|
||||
j, _ = call(s, rd, "getblocktemplate", [{}], keepalive=True)
|
||||
if j and isinstance(j.get("result"), dict):
|
||||
with SHARED_LOCK:
|
||||
SHARED["lpid"] = j["result"].get("longpollid")
|
||||
end_fin(s)
|
||||
log("bootstrap: %d hashes, %d zaddrs, lpid=%s" %
|
||||
(len(SHARED["hashes"]), len(SHARED["zaddrs"]), bool(SHARED["lpid"])))
|
||||
except Exception as e:
|
||||
log("bootstrap failed: %s" % e)
|
||||
|
||||
# ------------------------------------------------------------------ fd accounting
|
||||
|
||||
def count_fds(pid):
|
||||
if not pid:
|
||||
return (None, None, None)
|
||||
base = "/proc/%d/fd" % pid
|
||||
try:
|
||||
fds = os.listdir(base)
|
||||
except OSError:
|
||||
return (None, None, None)
|
||||
sock = files = 0
|
||||
for fd in fds:
|
||||
try:
|
||||
tgt = os.readlink(os.path.join(base, fd))
|
||||
except OSError:
|
||||
continue
|
||||
if tgt.startswith("socket:"):
|
||||
sock += 1
|
||||
elif tgt.startswith("/"):
|
||||
files += 1
|
||||
return (len(fds), sock, files)
|
||||
|
||||
def pid_alive(pid):
|
||||
if not pid:
|
||||
return True
|
||||
try:
|
||||
os.kill(pid, 0); return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
# ------------------------------------------------------------------ main
|
||||
|
||||
def reporter(pid):
|
||||
t0 = time.monotonic()
|
||||
while not STOP.is_set():
|
||||
for _ in range(10):
|
||||
if STOP.is_set():
|
||||
break
|
||||
time.sleep(0.1)
|
||||
w = ST.swap()
|
||||
tot, sk, fl = count_fds(pid)
|
||||
fdstr = ""
|
||||
if tot is not None:
|
||||
fdstr = " | fds=%d sock=%d file=%d" % (tot, sk, fl)
|
||||
log("t=%4ds ok=%d rpcerr=%d wq500=%d 4xx=%d other=%d | "
|
||||
"earlyRST=%d earlyFIN=%d srvRST=%d srvEOF=%d ctmo=%d connfail=%d | "
|
||||
"idmis=%d malf=%d | mined=%d halfopen=%d%s" % (
|
||||
int(time.monotonic() - t0),
|
||||
w["ok"], w["rpcerr"], w["wq500"], w["http4xx"], w["httpother"],
|
||||
w["early_rst"], w["early_fin"], w["srvreset"], w["srveof"],
|
||||
w["ctimeout"], w["connfail"], w["idmismatch"], w["malformed"],
|
||||
w["mined"], w["half_open"], fdstr))
|
||||
if pid and not pid_alive(pid):
|
||||
log("!!! NODE PROCESS %d IS GONE -- stopping harness" % pid)
|
||||
STOP.set()
|
||||
break
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--host", default="127.0.0.1")
|
||||
ap.add_argument("--port", type=int, default=19461)
|
||||
ap.add_argument("--user", default="ab")
|
||||
ap.add_argument("--password", default="abpass")
|
||||
ap.add_argument("--duration", type=int, default=300)
|
||||
ap.add_argument("--ctimeout", type=float, default=15.0)
|
||||
ap.add_argument("--normal", type=int, default=24)
|
||||
ap.add_argument("--impatient", type=int, default=10)
|
||||
ap.add_argument("--slowfire", type=int, default=3,
|
||||
help="paced clients that fire a genuinely-slow call then disconnect")
|
||||
ap.add_argument("--slowfire-pace", type=float, default=2.0)
|
||||
ap.add_argument("--halfopen", type=int, default=6)
|
||||
ap.add_argument("--pipeline", type=int, default=6)
|
||||
ap.add_argument("--garbage", type=int, default=4)
|
||||
ap.add_argument("--slowread", type=int, default=2)
|
||||
ap.add_argument("--longpoll", type=int, default=6)
|
||||
# burst-size must exceed (rpcthreads + rpcworkqueue) to overflow the queue even
|
||||
# when calls are fast; with -rpcthreads=8 -rpcworkqueue=4 that threshold is 12.
|
||||
ap.add_argument("--burst-threads", type=int, default=1)
|
||||
ap.add_argument("--burst-size", type=int, default=24)
|
||||
ap.add_argument("--burst-every", type=float, default=1.5)
|
||||
ap.add_argument("--mine", action="store_true", help="run a miner thread")
|
||||
ap.add_argument("--mine-interval", type=float, default=4.0)
|
||||
ap.add_argument("--pid", type=int, default=0, help="node pid, for /proc fd + liveness")
|
||||
ap.add_argument("--logfile", default="")
|
||||
args = ap.parse_args()
|
||||
|
||||
CFG.host = args.host; CFG.port = args.port
|
||||
CFG.auth = auth_header(args.user, args.password); CFG.ctimeout = args.ctimeout
|
||||
if args.logfile:
|
||||
LOGF[0] = open(args.logfile, "a")
|
||||
|
||||
log("hammer start host=%s port=%d duration=%ds pid=%s" %
|
||||
(CFG.host, CFG.port, args.duration, args.pid or "-"))
|
||||
bootstrap()
|
||||
|
||||
threads = []
|
||||
def spawn(fn, n, *a):
|
||||
for i in range(n):
|
||||
t = threading.Thread(target=fn, args=(i,) + a, daemon=True)
|
||||
t.start(); threads.append(t)
|
||||
|
||||
spawn(client_normal, args.normal)
|
||||
spawn(client_impatient, args.impatient)
|
||||
spawn(client_slowfire, args.slowfire, args.slowfire_pace)
|
||||
spawn(client_half_open, args.halfopen)
|
||||
spawn(client_pipeline, args.pipeline)
|
||||
spawn(client_garbage, args.garbage)
|
||||
spawn(client_slowread, args.slowread)
|
||||
spawn(client_longpoll, args.longpoll)
|
||||
for i in range(args.burst_threads):
|
||||
t = threading.Thread(target=client_burst, args=(i, args.burst_size, args.burst_every), daemon=True)
|
||||
t.start(); threads.append(t)
|
||||
if args.mine:
|
||||
t = threading.Thread(target=miner, args=(args.mine_interval,), daemon=True)
|
||||
t.start(); threads.append(t)
|
||||
|
||||
rt = threading.Thread(target=reporter, args=(args.pid,), daemon=True)
|
||||
rt.start()
|
||||
|
||||
end = time.monotonic() + args.duration
|
||||
try:
|
||||
while time.monotonic() < end and not STOP.is_set():
|
||||
time.sleep(0.2)
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
STOP.set()
|
||||
time.sleep(1.0)
|
||||
|
||||
log("=== TOTALS ===")
|
||||
for k in KEYS:
|
||||
if ST.total[k]:
|
||||
log(" %-14s %d" % (k, ST.total[k]))
|
||||
log("anomalies: %d" % len(ST.anom))
|
||||
for t, kind, detail in ST.anom[:40]:
|
||||
log(" %s %s %s" % (time.strftime("%H:%M:%S", time.localtime(t)), kind, detail))
|
||||
if args.pid:
|
||||
log("node alive at end: %s" % pid_alive(args.pid))
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,29 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# RPC-driven P2P peer churn on the target (same mix as fdbug/p2p/rpcchurn.sh):
|
||||
# disconnectnode / addnode onetry / setban+clearbanned / kill -9 + restart of helper2.
|
||||
# usage: rpcchurn.sh <run dir> <seconds>
|
||||
source "$REPRO/common.sh"
|
||||
OUT=$1; DUR=${2:-300}
|
||||
DD_T=$OUT/target; DD_H1=$OUT/helper1; DD_H2=$OUT/helper2
|
||||
end=$(( $(date +%s) + DUR )); i=0; restarts=0
|
||||
while [ "$(date +%s)" -lt "$end" ] && [ ! -e "$OUT/STOP" ]; do
|
||||
i=$((i+1))
|
||||
case $(( i % 5 )) in
|
||||
0) clin "$DD_T" "$RPC_T" disconnectnode "127.0.0.1:$P2P_H1" >/dev/null 2>&1 ;;
|
||||
1) clin "$DD_T" "$RPC_T" addnode "127.0.0.1:$P2P_H1" onetry >/dev/null 2>&1 ;;
|
||||
2) clin "$DD_T" "$RPC_T" setban "127.0.0.1" add 1 >/dev/null 2>&1; sleep 0.3
|
||||
clin "$DD_T" "$RPC_T" clearbanned >/dev/null 2>&1
|
||||
clin "$DD_H2" "$RPC_H2" addnode "127.0.0.1:$P2P_T" onetry >/dev/null 2>&1 ;;
|
||||
3) pid=$(cat "$DD_H2/regtest/hushd.pid" 2>/dev/null)
|
||||
if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then
|
||||
kill -9 "$pid" 2>/dev/null
|
||||
for _ in $(seq 1 20); do kill -0 "$pid" 2>/dev/null || break; sleep 0.5; done
|
||||
start_helper "$DD_H2" "$P2P_H2" "$RPC_H2" "$OUT/logs/helper2.txt"
|
||||
waitrpc "$DD_H2" "$RPC_H2" 120 >/dev/null 2>&1
|
||||
restarts=$((restarts+1))
|
||||
fi ;;
|
||||
4) clin "$DD_T" "$RPC_T" addnode "127.0.0.1:$P2P_H2" onetry >/dev/null 2>&1 ;;
|
||||
esac
|
||||
sleep 0.7
|
||||
done
|
||||
echo "rpcchurn: $i cycles, $restarts helper2 restarts"
|
||||
@@ -1,113 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
scan_bdb.py -- find Berkeley DB pages written into files that must not contain them
|
||||
(blocks/blk*.dat, blocks/rev*.dat). Stand-in for the production incident tool.
|
||||
|
||||
BDB writes page P of a database at file offset P*pagesize through pwrite(fd, ...). If
|
||||
fd has been closed and its number reused by a block file, the page lands in the block
|
||||
file at exactly P*pagesize. So at every pagesize-aligned offset we decode the PAGE
|
||||
header (db_page.h, all little-endian on x86):
|
||||
0 u32 lsn.file 4 u32 lsn.offset 8 u32 pgno 12 u32 prev_pgno
|
||||
16 u32 next_pgno 20 u16 entries 22 u16 hf_offset
|
||||
24 u8 level 25 u8 type
|
||||
and a metadata page (pgno 0 / any meta type) also carries magic at 12 and pagesize at 20.
|
||||
|
||||
A hit needs ALL of:
|
||||
* pgno == offset // pagesize (the self-locating property)
|
||||
* type is a real BDB page type (1..16 except 0)
|
||||
* level consistent with type (overflow/meta/hash: 0; btree leaf: 1; internal: 2..32)
|
||||
* hf_offset <= pagesize; entries <= pagesize/2
|
||||
* for overflow (7): entries (refcount) >= 1 and 0 < hf_offset (data length) <= pagesize-26
|
||||
Strength: "STRONG" if lsn == (0,1) (pages rewritten by lsn_reset -- the production
|
||||
signature) or a meta page with a valid magic, else "weak".
|
||||
Also reports any aligned BDB meta magic (btree 0x053162, hash 0x061561, queue 0x042253,
|
||||
heap 0x074582) even when pgno does not match.
|
||||
|
||||
usage: scan_bdb.py [--pagesize 4096] [--all-offsets] files...
|
||||
exit 0 = clean, 2 = hits.
|
||||
"""
|
||||
import argparse, struct, sys
|
||||
|
||||
TYPES = {1: "duplicate", 2: "hash_unsorted", 3: "ibtree", 4: "irecno", 5: "lbtree", 6: "lrecno",
|
||||
7: "overflow", 8: "hashmeta", 9: "btreemeta", 10: "qammeta", 11: "qamdata", 12: "ldup",
|
||||
13: "hash", 14: "heapmeta", 15: "heap", 16: "iheap"}
|
||||
MAGIC = {0x053162: "btree", 0x061561: "hash", 0x042253: "queue", 0x074582: "heap"}
|
||||
META = {8, 9, 10, 14}
|
||||
|
||||
|
||||
def level_ok(t, lvl):
|
||||
if t in (5, 6, 12): # leaf pages
|
||||
return lvl == 1
|
||||
if t in (3, 4): # internal pages
|
||||
return 2 <= lvl <= 32
|
||||
return lvl == 0 # overflow / meta / hash / queue / heap
|
||||
|
||||
|
||||
def check(buf, off, ps):
|
||||
if len(buf) < 26:
|
||||
return None
|
||||
lf, lo, pgno, prv, nxt, ent, hfo, lvl, typ = struct.unpack_from("<IIIIIHHBB", buf, 0)
|
||||
if typ not in TYPES or pgno != off // ps or not level_ok(typ, lvl):
|
||||
return None
|
||||
if typ in META:
|
||||
magic, ver, psz = struct.unpack_from("<III", buf, 12)
|
||||
if magic not in MAGIC:
|
||||
return None
|
||||
return ("STRONG", f"{TYPES[typ]} magic={MAGIC[magic]} ver={ver} pagesize={psz} lsn=({lf},{lo})")
|
||||
if hfo > ps or ent > ps // 2:
|
||||
return None
|
||||
if typ == 7 and not (ent >= 1 and 0 < hfo <= ps - 26):
|
||||
return None
|
||||
strength = "STRONG" if (lf, lo) == (0, 1) else "weak"
|
||||
return (strength, f"{TYPES[typ]} lsn=({lf},{lo}) prev={prv} next={nxt} entries={ent} hf_offset={hfo} level={lvl}")
|
||||
|
||||
|
||||
def scan(path, ps):
|
||||
hits, magic_only = [], []
|
||||
with open(path, "rb") as f:
|
||||
off = 0
|
||||
while True:
|
||||
buf = f.read(ps)
|
||||
if not buf:
|
||||
break
|
||||
r = check(buf, off, ps)
|
||||
if r:
|
||||
hits.append((off, off // ps) + r)
|
||||
elif len(buf) >= 16:
|
||||
m = struct.unpack_from("<I", buf, 12)[0]
|
||||
if m in MAGIC and buf[25] in META:
|
||||
magic_only.append((off, MAGIC[m]))
|
||||
off += len(buf)
|
||||
return hits, magic_only, off
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--pagesize", type=int, default=4096)
|
||||
ap.add_argument("--quiet", action="store_true")
|
||||
ap.add_argument("files", nargs="+")
|
||||
a = ap.parse_args()
|
||||
total = strong = 0
|
||||
for p in a.files:
|
||||
try:
|
||||
hits, mo, size = scan(p, a.pagesize)
|
||||
except OSError as e:
|
||||
print(f"{p}: ERROR {e}")
|
||||
continue
|
||||
s = sum(1 for h in hits if h[2] == "STRONG")
|
||||
total += len(hits); strong += s
|
||||
if hits or mo:
|
||||
print(f"{p}: {len(hits)} BDB page(s) ({s} strong), {len(mo)} stray meta magic, size={size}")
|
||||
if not a.quiet:
|
||||
for h in hits[:20]:
|
||||
print(f" off={h[0]} pgno={h[1]} {h[2]} {h[3]}")
|
||||
for m in mo[:5]:
|
||||
print(f" meta-magic off={m[0]} {m[1]}")
|
||||
else:
|
||||
print(f"{p}: clean ({size} bytes)")
|
||||
print(f"TOTAL BDB pages found: {total} ({strong} strong)")
|
||||
return 2 if total else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,80 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""tally.py <fdguard.log> -- attribute every fdguard report to (type, closer function, target class)
|
||||
and, for EBADF/NON-OWNER reports, to the code that made the previous close of that fd number.
|
||||
|
||||
The point: show that the stray closes ORIGINATE in CNode code (net.cpp CloseSocketDisconnect /
|
||||
~CNode closing a number they already closed) and that libevent / BDB / CreateNodeFromAcceptedSocket /
|
||||
GetTransaction only appear as victims-turned-amplifiers downstream of that.
|
||||
"""
|
||||
import collections, re, sys
|
||||
|
||||
ORIGIN = ("_ZN5CNode21CloseSocketDisconnectEv", "_ZN5CNodeD1Ev", "_ZN5CNodeD2Ev", "_ZN11CNetCleanupD2Ev")
|
||||
NAMES = {"_ZN5CNode21CloseSocketDisconnectEv": "CNode::CloseSocketDisconnect (net.cpp:572)",
|
||||
"_ZN5CNodeD1Ev": "CNode::~CNode (net.cpp:2819)",
|
||||
"_ZN11CNetCleanupD2Ev": "CNetCleanup::~CNetCleanup (net.cpp:2465, shutdown)",
|
||||
"evhttp_connection_free": "libevent evhttp_connection_free (http.c:1244)",
|
||||
"__os_closehandle": "BDB __os_closehandle",
|
||||
"_Z28CreateNodeFromAcceptedSocketjbRK8CAddressS1_": "CreateNodeFromAcceptedSocket (net.cpp:1336)",
|
||||
"_Z14GetTransactionRK7uint256R12CTransactionRS_b": "GetTransaction fclose (main.cpp)",
|
||||
"epoll_nochangelist_add": "libevent epoll ADD", "epoll_nochangelist_del": "libevent epoll DEL/MOD"}
|
||||
|
||||
|
||||
def fn(frame_line):
|
||||
m = re.search(r"\+0x[0-9a-f]+ (\S+?)\+0x", frame_line)
|
||||
return m.group(1) if m else "?"
|
||||
|
||||
|
||||
def main(path):
|
||||
reps, cur = [], None
|
||||
for line in open(path, errors="replace"):
|
||||
if line.startswith("=== FDGUARD #"):
|
||||
cur = {"hdr": line.strip(), "lines": []}
|
||||
reps.append(cur)
|
||||
elif cur is not None:
|
||||
if line.startswith("=== END"):
|
||||
cur = None
|
||||
else:
|
||||
cur["lines"].append(line.rstrip("\n"))
|
||||
tally = collections.Counter()
|
||||
prev_tally = collections.Counter()
|
||||
db_blk = []
|
||||
for r in reps:
|
||||
typ = r["hdr"].split()[3]
|
||||
L = r["lines"]
|
||||
cls = next((re.search(r"class=(\S+)", l).group(1) for l in L if " target=" in l and "class=" in l), "?")
|
||||
try:
|
||||
i = next(k for k, l in enumerate(L) if l.strip() == "closer stack:")
|
||||
closer = fn(L[i + 1])
|
||||
except StopIteration:
|
||||
closer = "?"
|
||||
prev = "-"
|
||||
for k, l in enumerate(L):
|
||||
if "previous successful close of this fd number" in l:
|
||||
# first frame line after the target= line
|
||||
for l2 in L[k + 1:k + 4]:
|
||||
if l2.strip().startswith("#0"):
|
||||
prev = fn(l2)
|
||||
break
|
||||
break
|
||||
tally[(typ, NAMES.get(closer, closer), cls)] += 1
|
||||
if typ in ("EBADF-CLOSE", "NON-OWNER-CLOSE"):
|
||||
prev_tally[NAMES.get(prev, prev)] += 1
|
||||
if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk"):
|
||||
tgt = next((l.split("target=")[1].split()[0] for l in L if " target=" in l), "?")
|
||||
db_blk.append((r["hdr"].split()[2], r["hdr"].split()[4], NAMES.get(closer, closer), tgt.rsplit("/", 2)[-2:]))
|
||||
print(f"{len(reps)} reports in {path}")
|
||||
print("\n count type class closer")
|
||||
for (typ, closer, cls), n in tally.most_common():
|
||||
print(f"{n:6d} {typ:17s} {cls:6s} {closer}")
|
||||
print("\nEBADF + NON-OWNER reports by the code that made the PREVIOUS close of the same number:")
|
||||
for k, n in prev_tally.most_common():
|
||||
print(f"{n:6d} {k}")
|
||||
o = sum(n for (typ, c, cls), n in tally.items() if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk") and c.startswith(("CNode", "CNetCleanup")))
|
||||
a = sum(n for (typ, c, cls), n in tally.items() if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk")) - o
|
||||
print(f"\nNON-OWNER closes of wallet/BDB-log/block files: {o + a} (by CNode code directly: {o}; by an amplifier: {a})")
|
||||
for d in db_blk[:40]:
|
||||
print(" ", d[0], d[1], "closer=" + d[2], "target=" + "/".join(d[3]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main(sys.argv[1])
|
||||
@@ -1,147 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Raw TCP / TLS client churn against the target node's P2P port.
|
||||
|
||||
dragonxd P2P is TLS 1.3 only (wolfSSL), self-signed, no cert verification. The
|
||||
accept path is net.cpp CreateNodeFromAcceptedSocket -> tlsmanager.accept()
|
||||
(wolfSSL_accept), and teardown is CloseSocketDisconnect / ~CNode
|
||||
(wolfSSL_shutdown + wolfSSL_free + CloseSocket). This driver pokes the accept and
|
||||
TLS-teardown/error paths (SSL_read/SSL_write at net.cpp ~1005 and tlsmanager.cpp
|
||||
~660-680) by opening many short-lived connections that each misbehave:
|
||||
|
||||
rst : connect, then abort with RST (SO_LINGER 0) before any TLS bytes
|
||||
fin : connect, then close cleanly (FIN) before any TLS bytes
|
||||
garbage : connect, send non-ClientHello bytes, then drop
|
||||
stall : connect, send a few bytes, hold mid-handshake, drop
|
||||
tlsjunk : complete a real TLS 1.3 handshake (verify off), send junk, drop
|
||||
tlshalf : start TLS, send a partial ClientHello record, drop
|
||||
|
||||
Loopback only, against a port this harness started. --duration seconds.
|
||||
"""
|
||||
import argparse, random, socket, ssl, struct, sys, time, os
|
||||
|
||||
def mk():
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
s.settimeout(3.0)
|
||||
return s
|
||||
|
||||
def rst_close(s):
|
||||
try:
|
||||
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
|
||||
except Exception:
|
||||
pass
|
||||
try: s.close()
|
||||
except Exception: pass
|
||||
|
||||
def do_rst(host, port):
|
||||
s = mk()
|
||||
try:
|
||||
s.connect((host, port)); rst_close(s)
|
||||
except Exception:
|
||||
rst_close(s)
|
||||
|
||||
def do_fin(host, port):
|
||||
s = mk()
|
||||
try:
|
||||
s.connect((host, port)); s.close()
|
||||
except Exception:
|
||||
try: s.close()
|
||||
except Exception: pass
|
||||
|
||||
def do_garbage(host, port):
|
||||
s = mk()
|
||||
try:
|
||||
s.connect((host, port))
|
||||
s.sendall(os.urandom(random.randint(16, 400)))
|
||||
time.sleep(random.uniform(0.0, 0.3))
|
||||
except Exception:
|
||||
pass
|
||||
rst_close(s)
|
||||
|
||||
def do_stall(host, port):
|
||||
s = mk()
|
||||
try:
|
||||
s.connect((host, port))
|
||||
# a couple of bytes that look like the start of a TLS record, then wait
|
||||
s.sendall(b"\x16\x03\x01")
|
||||
time.sleep(random.uniform(0.5, 2.0))
|
||||
except Exception:
|
||||
pass
|
||||
rst_close(s)
|
||||
|
||||
def _ctx():
|
||||
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
|
||||
ctx.check_hostname = False
|
||||
ctx.verify_mode = ssl.CERT_NONE
|
||||
try:
|
||||
ctx.minimum_version = ssl.TLSVersion.TLSv1_3
|
||||
except Exception:
|
||||
pass
|
||||
return ctx
|
||||
|
||||
def do_tlsjunk(host, port):
|
||||
raw = mk()
|
||||
try:
|
||||
raw.connect((host, port))
|
||||
tls = _ctx().wrap_socket(raw, server_hostname=None)
|
||||
# handshake completed; the node now expects a P2P version message.
|
||||
# send junk inside the TLS channel, then tear down.
|
||||
try:
|
||||
tls.sendall(os.urandom(random.randint(8, 128)))
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(random.uniform(0.0, 0.4))
|
||||
try:
|
||||
tls.close()
|
||||
except Exception:
|
||||
rst_close(raw)
|
||||
except Exception:
|
||||
rst_close(raw)
|
||||
|
||||
def do_tlshalf(host, port):
|
||||
s = mk()
|
||||
try:
|
||||
s.connect((host, port))
|
||||
# a well-formed TLS record header claiming a long handshake, but we send
|
||||
# only part of it, then drop -> server sits in wolfSSL_accept WANT_READ.
|
||||
body = os.urandom(40)
|
||||
rec = b"\x16\x03\x03" + struct.pack(">H", 512) + body
|
||||
s.sendall(rec)
|
||||
time.sleep(random.uniform(0.3, 1.5))
|
||||
except Exception:
|
||||
pass
|
||||
rst_close(s)
|
||||
|
||||
HANDLERS = [do_rst, do_fin, do_garbage, do_stall, do_tlsjunk, do_tlshalf]
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--host", default="127.0.0.1")
|
||||
ap.add_argument("--port", type=int, required=True)
|
||||
ap.add_argument("--duration", type=float, default=300.0)
|
||||
ap.add_argument("--concurrency", type=int, default=6)
|
||||
ap.add_argument("--rate", type=float, default=0.03, help="sleep between bursts")
|
||||
args = ap.parse_args()
|
||||
|
||||
end = time.time() + args.duration
|
||||
counts = {h.__name__: 0 for h in HANDLERS}
|
||||
import threading
|
||||
lock = threading.Lock()
|
||||
|
||||
def worker():
|
||||
while time.time() < end:
|
||||
h = random.choice(HANDLERS)
|
||||
h(args.host, args.port)
|
||||
with lock:
|
||||
counts[h.__name__] += 1
|
||||
time.sleep(random.uniform(0.0, args.rate))
|
||||
|
||||
ts = [threading.Thread(target=worker, daemon=True) for _ in range(args.concurrency)]
|
||||
for t in ts: t.start()
|
||||
for t in ts: t.join()
|
||||
|
||||
total = sum(counts.values())
|
||||
print(f"tcpchurn: {total} connections in {args.duration:.0f}s -> {counts}", flush=True)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,15 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Build the fdguard LD_PRELOAD shim and its self-test program.
|
||||
# ./build.sh -> libfdguard.so, fdguard_test
|
||||
set -e
|
||||
cd "$(dirname "$0")"
|
||||
CC=${CC:-gcc}
|
||||
$CC -O2 -g -fPIC -shared -Wall -Wextra -Wno-unused-parameter -fvisibility=default \
|
||||
-o libfdguard.so.tmp fdguard.c -ldl -lpthread
|
||||
# rename, never overwrite in place: a running process may have the old one mapped
|
||||
mv -f libfdguard.so.tmp libfdguard.so
|
||||
# the test program keeps its symbols (no -rdynamic on purpose: like dragonxd,
|
||||
# owner detection must work from .symtab alone) and frame pointers for clarity
|
||||
$CC -O1 -g -fno-omit-frame-pointer -fno-inline -Wall -o fdguard_test.tmp fdguard_test.c -lpthread
|
||||
mv -f fdguard_test.tmp fdguard_test
|
||||
echo "built: $(pwd)/libfdguard.so $(pwd)/fdguard_test"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,206 +0,0 @@
|
||||
/*
|
||||
* fdguard_test.c -- exercises libfdguard.so.
|
||||
*
|
||||
* The function names imitate the real owners inside dragonxd so the shim's
|
||||
* built-in domain patterns apply unchanged:
|
||||
* __os_openhandle / __os_closehandle Berkeley DB os layer (domain bdb)
|
||||
* OpenDiskFile / FlushBlockFile block-file code (domain blk)
|
||||
* ConnectSocketDirectly / CloseSocket P2P sockets (domain p2p)
|
||||
* foreign_stale_close / foreign_* somebody who holds a stale number
|
||||
*
|
||||
* usage: fdguard_test <dir> <case> (see main() for the list)
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <pthread.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/epoll.h>
|
||||
#include <sys/socket.h>
|
||||
#include <sys/syscall.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#define NI __attribute__((noinline))
|
||||
static char D[512];
|
||||
static char *P(const char *rel) { static char b[4][700]; static int i; i = (i + 1) & 3; snprintf(b[i], sizeof b[i], "%s/%s", D, rel); return b[i]; }
|
||||
static void say(const char *m) { fprintf(stderr, "[test] %s\n", m); }
|
||||
|
||||
/* --- "Berkeley DB" --- */
|
||||
NI int __os_openhandle(const char *path) { return open(path, O_RDWR | O_CREAT, 0600); }
|
||||
NI int __os_closehandle(int fd) { return close(fd); }
|
||||
/* --- "block files" --- */
|
||||
NI FILE *OpenDiskFile(const char *path) { FILE *f = fopen(path, "rb+"); if (!f) f = fopen(path, "wb+"); return f; }
|
||||
NI int FlushBlockFile(FILE *f) { fflush(f); return fclose(f); }
|
||||
/* --- "P2P" --- */
|
||||
NI int ConnectSocketDirectly(void) { return socket(AF_INET, SOCK_STREAM, 0); }
|
||||
NI int CloseSocket(int *s) { int r = close(*s); if (r) fprintf(stderr, "[test] Socket close failed: %d. Error: %s\n", *s, strerror(errno)); return r; }
|
||||
/* --- foreigners --- */
|
||||
NI int foreign_stale_close(int fd) { return close(fd); }
|
||||
NI int foreign_stale_fclose(FILE *f) { return fclose(f); }
|
||||
NI int foreign_dup2(int fd) { int dn = open("/dev/null", O_RDWR); int r = dup2(dn, fd); close(dn); return r; }
|
||||
NI int foreign_close_range(int fd) { return close_range((unsigned)fd, (unsigned)fd, 0); }
|
||||
NI void foreign_closefrom(int fd) { closefrom(fd); }
|
||||
/* --- plain code that opens and closes in the same function --- */
|
||||
NI int copy_like(const char *src) { int fd = open(src, O_RDONLY); char b[16]; if (fd >= 0) { (void)!read(fd, b, sizeof b); close(fd); } return fd; }
|
||||
NI int plain_open(const char *p) { return open(p, O_RDWR | O_CREAT, 0600); }
|
||||
NI int plain_close(int fd) { return close(fd); }
|
||||
|
||||
static int fd_valid(int fd) { return fcntl(fd, F_GETFD) != -1; }
|
||||
|
||||
static int c_double(void)
|
||||
{
|
||||
int fd = plain_open(P("plain.txt"));
|
||||
plain_close(fd);
|
||||
int r = plain_close(fd); /* EBADF */
|
||||
return r == -1 && errno == EBADF ? 0 : 1;
|
||||
}
|
||||
static int c_dbowner(void) { int fd = __os_openhandle(P("regtest/wallet.dat")); int r = __os_closehandle(fd); return r; }
|
||||
static int c_dbforeign(void)
|
||||
{
|
||||
int fd = __os_openhandle(P("regtest/wallet.dat"));
|
||||
foreign_stale_close(fd); /* HIT */
|
||||
int r = __os_closehandle(fd); /* BDB's own close then fails: EBADF (unless deny) */
|
||||
(void)r;
|
||||
return 0;
|
||||
}
|
||||
static int c_deny(void)
|
||||
{
|
||||
int fd = __os_openhandle(P("regtest/wallet.dat"));
|
||||
foreign_stale_close(fd); /* HIT + DENIED */
|
||||
if (!fd_valid(fd)) { say("deny FAILED: fd closed"); return 1; }
|
||||
if (write(fd, "x", 1) != 1) { say("deny FAILED: write"); return 1; }
|
||||
return __os_closehandle(fd);
|
||||
}
|
||||
static int c_blkowner(void) { FILE *f = OpenDiskFile(P("regtest/blocks/blk00000.dat")); fputs("blk", f); return FlushBlockFile(f); }
|
||||
static int c_blkraw(void)
|
||||
{
|
||||
FILE *f = OpenDiskFile(P("regtest/blocks/blk00001.dat"));
|
||||
foreign_stale_close(fileno(f)); /* HIT: raw close under a live FILE */
|
||||
FlushBlockFile(f); /* EBADF from fclose */
|
||||
return 0;
|
||||
}
|
||||
static int c_stalefile(void)
|
||||
{
|
||||
FILE *f = OpenDiskFile(P("regtest/blocks/blk00002.dat"));
|
||||
int n = fileno(f);
|
||||
foreign_stale_close(n); /* HIT #1 */
|
||||
int w = __os_openhandle(P("regtest/wallet.dat"));
|
||||
if (w != n) fprintf(stderr, "[test] note: wallet got fd %d, FILE had %d\n", w, n);
|
||||
FlushBlockFile(f); /* HIT #2: fclose of a stale FILE whose number is now wallet.dat */
|
||||
if (w == n) __os_closehandle(w); /* EBADF: BDB lost its fd */
|
||||
return 0;
|
||||
}
|
||||
static int c_reuse(void)
|
||||
{
|
||||
int s = ConnectSocketDirectly();
|
||||
int keep = s;
|
||||
CloseSocket(&s); /* legit */
|
||||
int w = __os_openhandle(P("regtest/wallet.dat")); /* reuses the number */
|
||||
if (w != keep) fprintf(stderr, "[test] note: wallet got fd %d, socket was %d\n", w, keep);
|
||||
s = keep;
|
||||
CloseSocket(&s); /* HIT: stale socket number now = wallet.dat */
|
||||
__os_closehandle(w); /* EBADF */
|
||||
return 0;
|
||||
}
|
||||
static int c_dup2(void) { int w = __os_openhandle(P("regtest/wallet.dat")); foreign_dup2(w); __os_closehandle(w); return 0; }
|
||||
static int c_range(void)
|
||||
{
|
||||
int w = __os_openhandle(P("regtest/wallet.dat"));
|
||||
foreign_close_range(w); /* HIT */
|
||||
w = __os_openhandle(P("regtest/database/log.0000000001"));
|
||||
int hi = fcntl(w, F_DUPFD, 600); /* move it high so closefrom only hits it */
|
||||
close(w);
|
||||
foreign_closefrom(hi); /* HIT (domain mismatch: opened by test main via fcntl dup? -> fallback db) */
|
||||
return 0;
|
||||
}
|
||||
static int c_owner_misc(void)
|
||||
{
|
||||
/* nothing here may be flagged */
|
||||
copy_like(P("regtest/wallet.dat")); /* common-caller */
|
||||
int fd = __os_openhandle(P("regtest/blocks/blk00003.dat"));
|
||||
FILE *f = fdopen(fd, "rb+"); /* ownership moves to FILE */
|
||||
FlushBlockFile(f);
|
||||
int s = ConnectSocketDirectly(); CloseSocket(&s);
|
||||
fd = (int)syscall(SYS_openat, AT_FDCWD, P("regtest/wallet.dat"), O_RDWR); /* untracked open */
|
||||
__os_closehandle(fd); /* fallback:bdb */
|
||||
return 0;
|
||||
}
|
||||
static int c_fallback_hit(void)
|
||||
{
|
||||
int fd = (int)syscall(SYS_openat, AT_FDCWD, P("regtest/wallet.dat"), O_RDWR); /* untracked open */
|
||||
foreign_stale_close(fd); /* HIT via fallback rule */
|
||||
return 0;
|
||||
}
|
||||
static int c_epoll(void)
|
||||
{
|
||||
int ep = epoll_create1(0);
|
||||
int fd = plain_open(P("plain2.txt"));
|
||||
struct epoll_event ev = { .events = EPOLLIN };
|
||||
int r = epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev); /* EPERM: regular file */
|
||||
close(fd);
|
||||
r = epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev); /* EBADF */
|
||||
close(ep);
|
||||
return r == -1 ? 0 : 1;
|
||||
}
|
||||
static int c_selfclose(void)
|
||||
{
|
||||
/* close every fd from 3 to 1100 by number, like a buggy daemonizer */
|
||||
for (int i = 3; i < 1100; i++) close(i);
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* concurrency + micro-benchmark: open/close churn in many threads */
|
||||
static int nthr = 8, iters = 20000;
|
||||
static void *worker(void *arg)
|
||||
{
|
||||
long id = (long)arg;
|
||||
char p[700]; snprintf(p, sizeof p, "%s/thr%ld.txt", D, id);
|
||||
for (int i = 0; i < iters; i++) {
|
||||
int s = ConnectSocketDirectly(); CloseSocket(&s);
|
||||
int fd = plain_open(p); plain_close(fd);
|
||||
FILE *f = fopen(p, "r"); if (f) fclose(f);
|
||||
if ((i & 255) == 0) { int w = __os_openhandle(P("regtest/wallet.dat")); __os_closehandle(w); }
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
static int c_threads(void)
|
||||
{
|
||||
const char *e = getenv("T_ITERS"); if (e) iters = atoi(e);
|
||||
pthread_t t[64];
|
||||
struct timespec a, b; clock_gettime(CLOCK_MONOTONIC, &a);
|
||||
for (long i = 0; i < nthr; i++) pthread_create(&t[i], NULL, worker, (void *)i);
|
||||
for (int i = 0; i < nthr; i++) pthread_join(t[i], NULL);
|
||||
clock_gettime(CLOCK_MONOTONIC, &b);
|
||||
double ns = (b.tv_sec - a.tv_sec) * 1e9 + (b.tv_nsec - a.tv_nsec);
|
||||
long ops = (long)nthr * iters * 3;
|
||||
printf("threads=%d iters=%d ops=%ld (socket/open/fopen + close each) wall=%.3fs %.0f ns/op-pair/thread\n",
|
||||
nthr, iters, ops, ns / 1e9, ns * nthr / ops);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int main(int argc, char **argv)
|
||||
{
|
||||
if (argc < 3) { fprintf(stderr, "usage: %s <dir> <case>\n", argv[0]); return 2; }
|
||||
snprintf(D, sizeof D, "%s", argv[1]);
|
||||
const char *c = argv[2];
|
||||
if (!strcmp(c, "double")) return c_double();
|
||||
if (!strcmp(c, "dbowner")) return c_dbowner();
|
||||
if (!strcmp(c, "dbforeign")) return c_dbforeign();
|
||||
if (!strcmp(c, "deny")) return c_deny();
|
||||
if (!strcmp(c, "blkowner")) return c_blkowner();
|
||||
if (!strcmp(c, "blkraw")) return c_blkraw();
|
||||
if (!strcmp(c, "stalefile")) return c_stalefile();
|
||||
if (!strcmp(c, "reuse")) return c_reuse();
|
||||
if (!strcmp(c, "dup2")) return c_dup2();
|
||||
if (!strcmp(c, "range")) return c_range();
|
||||
if (!strcmp(c, "ownermisc")) return c_owner_misc();
|
||||
if (!strcmp(c, "fallback")) return c_fallback_hit();
|
||||
if (!strcmp(c, "epoll")) return c_epoll();
|
||||
if (!strcmp(c, "selfclose")) return c_selfclose();
|
||||
if (!strcmp(c, "threads")) return c_threads();
|
||||
fprintf(stderr, "unknown case %s\n", c);
|
||||
return 2;
|
||||
}
|
||||
@@ -1,97 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Self-test for libfdguard.so. Every case lists the report types it must
|
||||
# produce (exactly, in order) -- an empty list means "no report at all".
|
||||
# ./run_tests.sh (builds first)
|
||||
set -u
|
||||
cd "$(dirname "$0")"
|
||||
./build.sh >/dev/null || { echo "build failed"; exit 1; }
|
||||
SO=$PWD/libfdguard.so
|
||||
T=$PWD/t
|
||||
rm -rf "$T"; mkdir -p "$T/regtest/blocks" "$T/regtest/database"
|
||||
pass=0; fail=0
|
||||
|
||||
run() { # run <case> [ENV=VAL ...] -> sets RC, LOG
|
||||
local c=$1; shift
|
||||
LOG=$T/$c${SUFFIX:-}.log; rm -f "$LOG"
|
||||
env FDGUARD_EXE=fdguard_test FDGUARD_LOG="$LOG" "$@" LD_PRELOAD="$SO" ./fdguard_test "$T" "$c" >"$T/$c${SUFFIX:-}.out" 2>&1
|
||||
RC=$?
|
||||
}
|
||||
types() { grep '^=== FDGUARD' "$LOG" 2>/dev/null | awk '{print $4}' | tr '\n' ' ' | sed 's/ $//'; }
|
||||
reasons() { grep -o 'reason=[^ ]*' "$LOG" 2>/dev/null | sed 's/reason=//' | tr '\n' ' ' | sed 's/ $//'; }
|
||||
check() { # check <name> <cond-description> <actual> <expected>
|
||||
if [ "$3" == "$4" ]; then pass=$((pass+1)); printf 'PASS %-12s %s\n' "$1" "$2";
|
||||
else fail=$((fail+1)); printf 'FAIL %-12s %s\n got: [%s]\n want: [%s]\n' "$1" "$2" "$3" "$4"; fi
|
||||
}
|
||||
|
||||
# case | expected report types | expected reasons
|
||||
while IFS='|' read -r c want wantr; do
|
||||
[ -z "$c" ] && continue
|
||||
run "$c"
|
||||
check "$c" "reports" "$(types)" "$want"
|
||||
check "$c" "reasons" "$(reasons)" "$wantr"
|
||||
check "$c" "exit 0" "$RC" "0"
|
||||
done <<'EOF'
|
||||
double|EBADF-CLOSE|
|
||||
dbowner||
|
||||
dbforeign|NON-OWNER-CLOSE EBADF-CLOSE|DOMAIN-MISMATCH
|
||||
blkowner||
|
||||
blkraw|NON-OWNER-CLOSE EBADF-CLOSE|stdio:RAW-CLOSE-UNDER-LIVE-FILE
|
||||
stalefile|NON-OWNER-CLOSE NON-OWNER-CLOSE EBADF-CLOSE|stdio:RAW-CLOSE-UNDER-LIVE-FILE stdio:STALE-FILE
|
||||
reuse|NON-OWNER-CLOSE EBADF-CLOSE|DOMAIN-MISMATCH
|
||||
dup2|NON-OWNER-CLOSE|DOMAIN-MISMATCH
|
||||
range|NON-OWNER-CLOSE NON-OWNER-CLOSE NON-OWNER-CLOSE|DOMAIN-MISMATCH DOMAIN-MISMATCH fallback:DB-FILE-CLOSED-OUTSIDE-BDB
|
||||
ownermisc||
|
||||
fallback|NON-OWNER-CLOSE|fallback:DB-FILE-CLOSED-OUTSIDE-BDB
|
||||
epoll|EPOLL-CTL-EPERM EPOLL-CTL-EBADF|epoll_ctl(ADD)-failed epoll_ctl(ADD)-failed
|
||||
EOF
|
||||
|
||||
# the previous close of the reused number must be the first CloseSocket (proves the double close)
|
||||
run reuse
|
||||
check reuse "history shows 1st CloseSocket on a socket" \
|
||||
"$(grep -A2 'previous successful close' "$LOG" | head -3 | grep -c -e 'class=sock dom=p2p' -e 'CloseSocket+')" "2"
|
||||
# the stale FILE report must name wallet.dat as the victim and OpenDiskFile as the FILE's opener
|
||||
run stalefile
|
||||
check stalefile "victim=wallet.dat, FILE opened in OpenDiskFile" \
|
||||
"$(sed -n '/#2 NON-OWNER/,/=== END #2/p' "$LOG" | grep -m1 -c 'target=.*wallet.dat class=db') $(sed -n '/#2 NON-OWNER/,/=== END #2/p' "$LOG" | grep -A3 'which had been opened by: fopen' | grep -c 'OpenDiskFile+')" "1 1"
|
||||
|
||||
# the log fd survives a close-everything loop and the attempt is reported
|
||||
run selfclose
|
||||
check selfclose "log-fd close refused + reported" "$(grep -c 'CLOSE-OF-FDGUARD-LOG-FD' "$LOG")" "1"
|
||||
check selfclose "log still writable afterwards (final stats)" "$(grep -c '^S .* final' "$LOG")" "1"
|
||||
|
||||
# abort mode: first non-owner close -> SIGABRT (rc 134), report already written
|
||||
SUFFIX=_abort run dbforeign FDGUARD_ABORT=1
|
||||
check abort "rc=134 (SIGABRT)" "$RC" "134"
|
||||
check abort "report written before abort" "$(types)" "NON-OWNER-CLOSE"
|
||||
# abort=2 also aborts on EBADF
|
||||
SUFFIX=_abort2 run double FDGUARD_ABORT=2
|
||||
check abort2 "rc=134 on EBADF" "$RC" "134"
|
||||
|
||||
# deny mode: wallet fd survives the foreign close; no EBADF afterwards
|
||||
SUFFIX=_deny run deny FDGUARD_DENY=1
|
||||
check deny "exit 0 (fd still valid + writable)" "$RC" "0"
|
||||
check deny "reported as DENIED, no EBADF" "$(types) $(grep -c 'ACTION=DENIED' "$LOG")" "NON-OWNER-CLOSE 1"
|
||||
|
||||
# extra classes: guard=all with the owner cases must still be silent
|
||||
SUFFIX=_all run ownermisc FDGUARD_GUARD=all
|
||||
check guard-all "ownermisc silent with FDGUARD_GUARD=all" "$(types)" ""
|
||||
|
||||
# trace-all writes one T line per open/close
|
||||
SUFFIX=_trace run dbowner FDGUARD_TRACE_ALL=1
|
||||
check trace "T lines for open+close" "$(grep -c '^T ' "$LOG")" "2"
|
||||
|
||||
# passive in non-matching processes
|
||||
LOG=$T/passive.log; env FDGUARD_EXE=dragonxd FDGUARD_LOG="$LOG" LD_PRELOAD="$SO" ./fdguard_test "$T" dbforeign >/dev/null 2>&1
|
||||
check passive "no log when exe does not match" "$( [ -e "$LOG" ] && echo exists || echo none)" "none"
|
||||
|
||||
# concurrency: 8 threads x 5000 x (socket, open, fopen) churn, guard=all -> no reports
|
||||
SUFFIX=_thr run threads T_ITERS=5000 FDGUARD_GUARD=all
|
||||
check threads "no reports under 8-thread churn (guard=all)" "$(types)" ""
|
||||
check threads "stats: HITS=0 EBADF=0" "$(grep '^S .*final' "$LOG" | grep -o 'HITS=[0-9]* EBADF=[0-9]*')" "HITS=0 EBADF=0"
|
||||
|
||||
# symbolizer resolves file:line for the test binary
|
||||
check symbolize "addr2line resolved CloseSocket to fdguard_test.c" \
|
||||
"$(./symbolize.py "$T/reuse.log" | grep -m1 -o 'in CloseSocket at .*fdguard_test.c:[0-9]*' | sed 's/.*fdguard_test.c:[0-9]*/ok/')" "ok"
|
||||
|
||||
echo "== $pass passed, $fail failed"
|
||||
[ $fail -eq 0 ]
|
||||
@@ -1,212 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
symbolize.py -- turn fdguard log frames into function + file:line.
|
||||
|
||||
Every stack frame in an fdguard report looks like
|
||||
#3 0x5b96b433c991 /path/to/module+0x1991 c_dbforeign+0x2d [bdb]
|
||||
where "+0x1991" is the ELF virtual address inside that module (runtime address
|
||||
minus the module's load bias, so PIE executables and shared libraries need no
|
||||
further adjustment; for a non-PIE executable it equals the absolute address).
|
||||
Frames are return addresses, so we ask addr2line about vaddr-1 (the call
|
||||
instruction), with -f -C -i (function, demangle, inline chain).
|
||||
|
||||
backtrace_symbols_fd lines "module(+0x1991)[0x...]" are symbolized too.
|
||||
|
||||
usage:
|
||||
symbolize.py [options] LOG [LOG...] annotated log on stdout
|
||||
symbolize.py --summary LOG one line per distinct report
|
||||
options:
|
||||
--map OLD=NEW read module OLD from path NEW instead (repeatable),
|
||||
e.g. --map /path/to/deployed/dragonxd=./dragonxd.unstripped
|
||||
--addr2line PATH addr2line binary (default: addr2line)
|
||||
--only-reports drop T/S lines (trace + stats) from the output
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from collections import defaultdict, OrderedDict
|
||||
|
||||
FRAME_RE = re.compile(r'^(\s*#\d+ 0x[0-9a-f]+ )(\S+)\+0x([0-9a-f]+)(.*)$')
|
||||
BSFD_RE = re.compile(r'^(\S+)\(\+0x([0-9a-f]+)\)\[0x[0-9a-f]+\]\s*$')
|
||||
HDR_RE = re.compile(r'^=== FDGUARD #(\d+) (\S+)')
|
||||
|
||||
|
||||
def parse_args(argv):
|
||||
opts = {'map': {}, 'addr2line': 'addr2line', 'summary': False, 'only': False, 'logs': []}
|
||||
it = iter(argv)
|
||||
for a in it:
|
||||
if a == '--map':
|
||||
old, new = next(it).split('=', 1)
|
||||
opts['map'][old] = new
|
||||
elif a.startswith('--map='):
|
||||
old, new = a[6:].split('=', 1)
|
||||
opts['map'][old] = new
|
||||
elif a == '--addr2line':
|
||||
opts['addr2line'] = next(it)
|
||||
elif a == '--summary':
|
||||
opts['summary'] = True
|
||||
elif a == '--only-reports':
|
||||
opts['only'] = True
|
||||
elif a in ('-h', '--help'):
|
||||
print(__doc__)
|
||||
sys.exit(0)
|
||||
else:
|
||||
opts['logs'].append(a)
|
||||
if not opts['logs']:
|
||||
opts['logs'] = ['-']
|
||||
return opts
|
||||
|
||||
|
||||
def read_lines(paths):
|
||||
out = []
|
||||
for p in paths:
|
||||
f = sys.stdin if p == '-' else open(p, errors='replace')
|
||||
out.extend(f.read().splitlines())
|
||||
if f is not sys.stdin:
|
||||
f.close()
|
||||
return out
|
||||
|
||||
|
||||
def resolve_module(mod, opts, known):
|
||||
if mod in opts['map']:
|
||||
return opts['map'][mod]
|
||||
if os.path.exists(mod):
|
||||
return mod
|
||||
base = os.path.basename(mod)
|
||||
for k in known:
|
||||
if os.path.basename(k) == base and os.path.exists(opts['map'].get(k, k)):
|
||||
return opts['map'].get(k, k)
|
||||
return None
|
||||
|
||||
|
||||
def run_addr2line(binary, addrs, a2l):
|
||||
"""addrs: list of int vaddrs (already -1'd). returns {addr: [(func, loc), ...]}"""
|
||||
res = {}
|
||||
if not addrs:
|
||||
return res
|
||||
uniq = sorted(set(addrs))
|
||||
try:
|
||||
p = subprocess.run([a2l, '-a', '-f', '-C', '-i', '-e', binary] + ['0x%x' % a for a in uniq],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, check=False)
|
||||
except FileNotFoundError:
|
||||
return res
|
||||
cur = None
|
||||
lines = p.stdout.splitlines()
|
||||
i = 0
|
||||
while i < len(lines):
|
||||
ln = lines[i]
|
||||
if re.match(r'^0x[0-9a-f]+$', ln):
|
||||
cur = int(ln, 16)
|
||||
res[cur] = []
|
||||
i += 1
|
||||
continue
|
||||
func = ln
|
||||
loc = lines[i + 1] if i + 1 < len(lines) else '??:0'
|
||||
if cur is not None:
|
||||
res[cur].append((func, loc))
|
||||
i += 2
|
||||
return res
|
||||
|
||||
|
||||
def main():
|
||||
opts = parse_args(sys.argv[1:])
|
||||
lines = read_lines(opts['logs'])
|
||||
|
||||
if opts['summary']:
|
||||
summary(lines)
|
||||
return
|
||||
|
||||
# pass 1: collect modules + addresses
|
||||
want = defaultdict(set)
|
||||
known = OrderedDict()
|
||||
for ln in lines:
|
||||
m = FRAME_RE.match(ln)
|
||||
if m:
|
||||
known[m.group(2)] = 1
|
||||
want[m.group(2)].add(max(int(m.group(3), 16) - 1, 0))
|
||||
for ln in lines:
|
||||
m = BSFD_RE.match(ln)
|
||||
if m:
|
||||
want[m.group(1)].add(max(int(m.group(2), 16) - 1, 0))
|
||||
|
||||
table = {}
|
||||
for mod, addrs in want.items():
|
||||
path = resolve_module(mod, opts, known)
|
||||
if not path:
|
||||
continue
|
||||
table[mod] = run_addr2line(path, list(addrs), opts['addr2line'])
|
||||
|
||||
def fmt(mod, vaddr):
|
||||
r = table.get(mod, {}).get(vaddr)
|
||||
if not r:
|
||||
return []
|
||||
out = []
|
||||
for j, (func, loc) in enumerate(r):
|
||||
if func == '??' and loc.startswith('??'):
|
||||
continue
|
||||
# addr2line -i lists the innermost (inlined) frame first
|
||||
tag = 'inlined:' if j < len(r) - 1 else 'in'
|
||||
out.append('%s %s at %s' % (tag, func, loc))
|
||||
return out
|
||||
|
||||
w = sys.stdout.write
|
||||
for ln in lines:
|
||||
if opts['only'] and (ln.startswith('T ') or ln.startswith('S ')):
|
||||
continue
|
||||
w(ln + '\n')
|
||||
m = FRAME_RE.match(ln)
|
||||
if m:
|
||||
v = max(int(m.group(3), 16) - 1, 0)
|
||||
for s in fmt(m.group(2), v):
|
||||
w(' -> ' + s + '\n')
|
||||
continue
|
||||
m = BSFD_RE.match(ln)
|
||||
if m:
|
||||
v = max(int(m.group(2), 16) - 1, 0)
|
||||
for s in fmt(m.group(1), v):
|
||||
w(' -> ' + s + '\n')
|
||||
|
||||
|
||||
def summary(lines):
|
||||
"""one line per (type, reason, class, first exe closer frame)"""
|
||||
groups = OrderedDict()
|
||||
cur = None
|
||||
in_closer = False
|
||||
for ln in lines:
|
||||
h = HDR_RE.match(ln)
|
||||
if h:
|
||||
cur = {'type': h.group(2), 'reason': '', 'cls': '', 'top': '', 'call': ''}
|
||||
in_closer = False
|
||||
continue
|
||||
if cur is None:
|
||||
continue
|
||||
s = ln.strip()
|
||||
if s.startswith('call:'):
|
||||
cur['call'] = s.split('(')[0][6:]
|
||||
elif s.startswith('target=') and ' class=' in s and not cur['cls']:
|
||||
cur['cls'] = s.rsplit('class=', 1)[-1]
|
||||
elif s.startswith('verdict='):
|
||||
cur['reason'] = s.split('reason=', 1)[-1].split(' ')[0]
|
||||
elif s == 'closer stack:':
|
||||
in_closer = True
|
||||
elif in_closer and FRAME_RE.match(ln):
|
||||
parts = s.split(' ')
|
||||
if len(parts) >= 4 and not cur['top'] and '/libc.so' not in parts[2] and '/libstdc++' not in parts[2]:
|
||||
cur['top'] = parts[3]
|
||||
elif in_closer and not s.startswith('#'):
|
||||
in_closer = False
|
||||
if s.startswith('=== END'):
|
||||
k = (cur['type'], cur['call'], cur['reason'], cur['cls'], cur['top'])
|
||||
groups[k] = groups.get(k, 0) + 1
|
||||
cur = None
|
||||
print('%6s %-22s %-12s %-6s %-40s %s' % ('count', 'type', 'call', 'class', 'reason', 'first closer frame'))
|
||||
for (t, c, r, cl, top), n in groups.items():
|
||||
print('%6d %-22s %-12s %-6s %-40s %s' % (n, t[:22], c, cl, r[:40], top))
|
||||
stats = [ln for ln in lines if ln.startswith('S ')]
|
||||
if stats:
|
||||
print('\nlast stats line:\n ' + stats[-1])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -15,12 +15,8 @@ hush_gtest_SOURCES = \
|
||||
gtest/test_hdtransparent.cpp \
|
||||
gtest/test_mnemonic_compat.cpp \
|
||||
gtest/test_stratum_jobid.cpp \
|
||||
gtest/test_wallet_notedata.cpp \
|
||||
gtest/test_net_closesocket.cpp \
|
||||
gtest/test_fd_defense.cpp \
|
||||
gtest/test_wallet_dbenv.cpp \
|
||||
gtest/test_wallet_notecache.cpp \
|
||||
gtest/test_wallet_history.cpp
|
||||
gtest/test_wallet_scan_notes.cpp \
|
||||
gtest/test_operation_status.cpp
|
||||
|
||||
hush_gtest_CPPFLAGS = $(AM_CPPFLAGS) -DMULTICORE -fopenmp -DBINARY_OUTPUT -DCURVE_ALT_BN128 -DSTATIC $(BITCOIN_INCLUDES)
|
||||
hush_gtest_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
|
||||
|
||||
@@ -32,7 +32,7 @@ using namespace std;
|
||||
|
||||
static boost::uuids::random_generator uuidgen;
|
||||
|
||||
static std::map<OperationStatus, std::string> OperationStatusMap = {
|
||||
static const std::map<OperationStatus, std::string> OperationStatusMap = {
|
||||
{OperationStatus::READY, "queued"},
|
||||
{OperationStatus::EXECUTING, "executing"},
|
||||
{OperationStatus::CANCELLED, "cancelled"},
|
||||
@@ -194,7 +194,7 @@ UniValue AsyncRPCOperation::getStatus() const {
|
||||
OperationStatus status = this->getState();
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
obj.push_back(Pair("id", this->id_));
|
||||
obj.push_back(Pair("status", OperationStatusMap[status]));
|
||||
obj.push_back(Pair("status", OperationStatusMap.at(status)));
|
||||
obj.push_back(Pair("creation_time", this->creation_time_));
|
||||
// TODO: Issue #1354: There may be other useful metadata to return to the user.
|
||||
UniValue err = this->getError();
|
||||
@@ -218,5 +218,5 @@ UniValue AsyncRPCOperation::getStatus() const {
|
||||
*/
|
||||
std::string AsyncRPCOperation::getStateAsString() const {
|
||||
OperationStatus status = this->getState();
|
||||
return OperationStatusMap[status];
|
||||
return OperationStatusMap.at(status);
|
||||
}
|
||||
|
||||
@@ -129,14 +129,18 @@ std::shared_ptr<AsyncRPCOperation> AsyncRPCQueue::getOperationForId(AsyncRPCOper
|
||||
|
||||
/**
|
||||
* Return the operation for a given operation id and then remove the operation from internal storage.
|
||||
* The lookup and the erase happen under one hold of lock_, so when several threads pop the same id
|
||||
* exactly one of them receives the operation and the others get a null pointer.
|
||||
*/
|
||||
std::shared_ptr<AsyncRPCOperation> AsyncRPCQueue::popOperationForId(AsyncRPCOperationId id) {
|
||||
std::shared_ptr<AsyncRPCOperation> ptr = getOperationForId(id);
|
||||
if (ptr) {
|
||||
std::lock_guard<std::mutex> guard(lock_);
|
||||
std::shared_ptr<AsyncRPCOperation> ptr; // declared before guard, so it is released after unlock
|
||||
std::lock_guard<std::mutex> guard(lock_);
|
||||
AsyncRPCOperationMap::iterator iter = operation_map_.find(id);
|
||||
if (iter != operation_map_.end()) {
|
||||
ptr = iter->second;
|
||||
// Note: if the id still exists in the operationIdQueue, when it gets processed by a worker
|
||||
// there will no operation in the map to execute, so nothing will happen.
|
||||
operation_map_.erase(id);
|
||||
operation_map_.erase(iter);
|
||||
}
|
||||
return ptr;
|
||||
}
|
||||
|
||||
@@ -1,83 +0,0 @@
|
||||
// Copyright (c) 2016-2024 The Hush developers
|
||||
// Distributed under the GPLv3 software license, see the accompanying
|
||||
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <atomic>
|
||||
#include <errno.h>
|
||||
#include <stdexcept>
|
||||
#include <stdio.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include <boost/filesystem.hpp>
|
||||
#include <boost/thread.hpp>
|
||||
|
||||
#include "chainparams.h"
|
||||
#include "init.h"
|
||||
#include "main.h"
|
||||
#include "noui.h"
|
||||
#include "util.h"
|
||||
#include "validationinterface.h"
|
||||
|
||||
extern std::atomic<bool> fRequestShutdown;
|
||||
|
||||
// A block file opened for reading must not be writable through its fd: a stray write through a reused fd number
|
||||
// (the wolfSSL CloseSocket macro let the wallet's BDB fd be closed behind its back) used to land in it.
|
||||
TEST(FdDefense, ReadOnlyBlockFileIsNotWritable) {
|
||||
SelectParams(CBaseChainParams::MAIN);
|
||||
const boost::filesystem::path dir = boost::filesystem::temp_directory_path() / boost::filesystem::unique_path("fddefense-%%%%%%%%");
|
||||
const bool hadDatadir = mapArgs.count("-datadir") > 0;
|
||||
const std::string oldDatadir = hadDatadir ? mapArgs["-datadir"] : "";
|
||||
boost::filesystem::create_directories(dir);
|
||||
mapArgs["-datadir"] = dir.string();
|
||||
ClearDatadirCache();
|
||||
|
||||
const CDiskBlockPos pos(0, 0);
|
||||
const boost::filesystem::path blk = GetBlockPosFilename(pos, "blk");
|
||||
boost::filesystem::create_directories(blk.parent_path());
|
||||
{
|
||||
FILE* f = fopen(blk.string().c_str(), "wb");
|
||||
ASSERT_TRUE(f != nullptr);
|
||||
fputs("block bytes", f);
|
||||
fclose(f);
|
||||
}
|
||||
|
||||
FILE* ro = OpenBlockFile(pos, true);
|
||||
ASSERT_TRUE(ro != nullptr);
|
||||
char c = 0;
|
||||
EXPECT_EQ(1u, fread(&c, 1, 1, ro)) << "still readable";
|
||||
errno = 0;
|
||||
EXPECT_EQ(-1, pwrite(fileno(ro), "X", 1, 0)) << "a write through a read-only block file's fd must fail";
|
||||
EXPECT_EQ(EBADF, errno);
|
||||
fclose(ro);
|
||||
|
||||
FILE* rw = OpenBlockFile(pos, false);
|
||||
ASSERT_TRUE(rw != nullptr);
|
||||
EXPECT_EQ(1, pwrite(fileno(rw), "b", 1, 0)) << "the writable open still writes";
|
||||
fclose(rw);
|
||||
|
||||
if (hadDatadir) mapArgs["-datadir"] = oldDatadir; else mapArgs.erase("-datadir");
|
||||
ClearDatadirCache();
|
||||
boost::filesystem::remove_all(dir);
|
||||
}
|
||||
|
||||
// A wallet notification that throws (a wallet DB failure) requests a clean shutdown instead of escaping the
|
||||
// notifier thread into std::terminate; a successful one doesn't; thread interruption still propagates.
|
||||
TEST(FdDefense, WalletNotifyFailureShutsDownCleanly) {
|
||||
noui_connect(); // the daemon connects these UI signals at startup; the error path shows a message box
|
||||
fRequestShutdown = false;
|
||||
EXPECT_TRUE(NotifyWalletsOrShutdown([] {}));
|
||||
EXPECT_FALSE(ShutdownRequested());
|
||||
|
||||
EXPECT_FALSE(NotifyWalletsOrShutdown([] { throw std::runtime_error("CWalletDB::ListAccountCreditDebit(): error scanning DB"); }));
|
||||
EXPECT_TRUE(ShutdownRequested());
|
||||
fRequestShutdown = false;
|
||||
|
||||
EXPECT_FALSE(NotifyWalletsOrShutdown([] { throw 42; }));
|
||||
EXPECT_TRUE(ShutdownRequested());
|
||||
fRequestShutdown = false;
|
||||
|
||||
EXPECT_THROW(NotifyWalletsOrShutdown([] { throw boost::thread_interrupted(); }), boost::thread_interrupted);
|
||||
EXPECT_FALSE(ShutdownRequested());
|
||||
}
|
||||
@@ -1,50 +0,0 @@
|
||||
// Copyright (c) 2016-2024 The Hush developers
|
||||
// Distributed under the GPLv3 software license, see the accompanying
|
||||
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <fcntl.h>
|
||||
#include <sys/socket.h>
|
||||
#include <unistd.h>
|
||||
|
||||
#include "net.h"
|
||||
#include "util/sock.h"
|
||||
|
||||
// Regression test for the wolfSSL CloseSocket macro (see net.h): closing a CNode's socket must invalidate
|
||||
// hSocket, so neither a repeated CloseSocketDisconnect nor ~CNode can close that fd number again once another
|
||||
// owner (the wallet's BDB files, a block file, an RPC socket) has been handed it.
|
||||
TEST(NetCloseSocket, ClosedNodeNeverClosesAReusedFdNumber) {
|
||||
int pair[2];
|
||||
ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_STREAM, 0, pair));
|
||||
const int number = pair[0];
|
||||
|
||||
CNode* node = new CNode(number, CAddress(), "", true); // inbound: sends no version message
|
||||
node->CloseSocketDisconnect();
|
||||
EXPECT_EQ(INVALID_SOCKET, node->hSocket) << "CloseSocketDisconnect left the closed fd number in hSocket";
|
||||
EXPECT_EQ(-1, fcntl(number, F_GETFD)) << "the node's socket wasn't closed";
|
||||
|
||||
// Another owner gets the freed number, as the kernel's lowest-free-fd rule does in a busy node.
|
||||
const int other = open("/dev/null", O_RDONLY);
|
||||
ASSERT_GE(other, 0);
|
||||
ASSERT_EQ(number, dup2(other, number));
|
||||
|
||||
node->CloseSocketDisconnect(); // e.g. a send error after the disconnect was already handled
|
||||
delete node; // ~CNode
|
||||
EXPECT_NE(-1, fcntl(number, F_GETFD)) << "a closed CNode closed the fd number's new owner";
|
||||
|
||||
close(number);
|
||||
close(other);
|
||||
close(pair[1]);
|
||||
}
|
||||
|
||||
// util/sock.cpp's CloseSocket is the one that must be called: it closes once and invalidates.
|
||||
TEST(NetCloseSocket, CloseSocketInvalidates) {
|
||||
int pair[2];
|
||||
ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_STREAM, 0, pair));
|
||||
SOCKET s = pair[0];
|
||||
EXPECT_TRUE(CloseSocket(s));
|
||||
EXPECT_EQ(INVALID_SOCKET, s);
|
||||
EXPECT_FALSE(CloseSocket(s)) << "a second close of an invalidated handle must be a no-op";
|
||||
close(pair[1]);
|
||||
}
|
||||
@@ -0,0 +1,936 @@
|
||||
// Copyright (c) 2016-2026 The Hush developers
|
||||
// Distributed under the GPLv3 software license, see the accompanying
|
||||
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
|
||||
//
|
||||
// z_getoperationstatus / z_getoperationresult / z_listoperationids after they stopped sorting
|
||||
// whole UniValues, and after z_getoperationstatus and z_listoperationids stopped taking
|
||||
// cs_main + cs_wallet. These tests show that:
|
||||
// * both status calls return byte for byte what the original algorithm (copied below from
|
||||
// c1040028e) returns, for queued, executing, successful, failed and cancelled operations,
|
||||
// with and without id filters, with unknown and duplicate ids, with more than 16
|
||||
// operations and with creation_time ties. std::sort reorders equal keys once there are
|
||||
// more than 16 of them, so the order of same-second operations is part of the contract;
|
||||
// * z_getoperationresult removes exactly the finished operations that it reports;
|
||||
// * z_getoperationstatus and z_listoperationids answer while another thread holds
|
||||
// cs_main + cs_wallet, and z_getoperationresult still waits for those locks;
|
||||
// * AsyncRPCQueue::popOperationForId hands an operation to exactly one of several concurrent
|
||||
// callers, so concurrent removers never report the same operation twice.
|
||||
//
|
||||
// No worker runs on the shared async queue in hush-gtest, so every operation keeps the state the
|
||||
// test gives it. Without pwalletMain the handlers throw "Method not found", and
|
||||
// z_getoperationresult locks pwalletMain->cs_wallet, so each test installs an empty in-memory
|
||||
// CWallet. These tests use TEST() and a TestEnv guard rather than a fixture, because a fixture
|
||||
// class cannot be named OperationStatus: that name is the operation state enum in
|
||||
// asyncrpcoperation.h.
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
#include <univalue.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <condition_variable>
|
||||
#include <ctime>
|
||||
#include <future>
|
||||
#include <map>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <set>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
#include "asyncrpcoperation.h"
|
||||
#include "asyncrpcqueue.h"
|
||||
#include "init.h"
|
||||
#include "main.h"
|
||||
#include "pubkey.h"
|
||||
#include "rpc/server.h"
|
||||
#include "sync.h"
|
||||
#include "wallet/wallet.h"
|
||||
|
||||
// Defined in wallet/rpcwallet.cpp (non-static, no header). Called directly so that removers can
|
||||
// run without the cs_main + cs_wallet that z_getoperationresult takes.
|
||||
extern UniValue z_getoperationstatus_IMPL(const UniValue& params, bool fRemoveFinishedOperations);
|
||||
|
||||
namespace {
|
||||
|
||||
// Bound for every wait that must succeed, so a regression fails instead of hanging.
|
||||
const std::chrono::milliseconds kGuardTimeout(5000);
|
||||
|
||||
const char* const kQueueNotEmpty = "the shared async queue already holds operations from elsewhere";
|
||||
const char* const kPoolFailed = "could not create operations with controlled creation_time ties";
|
||||
|
||||
enum FakeKind {
|
||||
FAKE_QUEUED = 0,
|
||||
FAKE_EXECUTING,
|
||||
FAKE_SUCCESS,
|
||||
FAKE_FAILED,
|
||||
FAKE_CANCELLED,
|
||||
FAKE_KINDS
|
||||
};
|
||||
|
||||
// An operation whose state is set by the test and never changes after it is queued: main() does
|
||||
// nothing, and nothing runs it anyway. getStatus() appends method/params exactly the way
|
||||
// z_sendmany, z_mergetoaddress and z_shieldcoinbase append their contextinfo_.
|
||||
class FixedStateOperation : public AsyncRPCOperation {
|
||||
public:
|
||||
virtual void main() {}
|
||||
|
||||
virtual UniValue getStatus() const {
|
||||
UniValue v = AsyncRPCOperation::getStatus();
|
||||
if (contextinfo_.isNull()) {
|
||||
return v;
|
||||
}
|
||||
|
||||
UniValue obj = v.get_obj();
|
||||
obj.push_back(Pair("method", "z_fixedstate"));
|
||||
obj.push_back(Pair("params", contextinfo_));
|
||||
return obj;
|
||||
}
|
||||
|
||||
// Only called before the operation is queued, like a real operation's constructor.
|
||||
void SetContext(const UniValue& context) {
|
||||
contextinfo_ = context;
|
||||
}
|
||||
|
||||
void ForceExecuting() {
|
||||
set_state(OperationStatus::EXECUTING);
|
||||
}
|
||||
|
||||
// Same order as the real operations: stop the clock, then publish the result and SUCCESS.
|
||||
void ForceSuccess(const UniValue& result) {
|
||||
start_execution_clock();
|
||||
stop_execution_clock();
|
||||
set_result(result);
|
||||
set_state(OperationStatus::SUCCESS);
|
||||
}
|
||||
|
||||
void ForceFailed(int code, const std::string& message) {
|
||||
set_error_code(code);
|
||||
set_error_message(message);
|
||||
set_state(OperationStatus::FAILED);
|
||||
}
|
||||
|
||||
private:
|
||||
UniValue contextinfo_;
|
||||
};
|
||||
|
||||
std::shared_ptr<FixedStateOperation> MakeOperation(int kind, int tag)
|
||||
{
|
||||
std::shared_ptr<FixedStateOperation> op(new FixedStateOperation());
|
||||
if (tag % 3 == 0) {
|
||||
UniValue context(UniValue::VOBJ);
|
||||
context.push_back(Pair("fromaddress", "zs1fixedstate"));
|
||||
context.push_back(Pair("minconf", tag));
|
||||
op->SetContext(context);
|
||||
}
|
||||
switch (kind) {
|
||||
case FAKE_QUEUED:
|
||||
break;
|
||||
case FAKE_EXECUTING:
|
||||
op->ForceExecuting();
|
||||
break;
|
||||
case FAKE_SUCCESS:
|
||||
if (tag % 2) {
|
||||
UniValue result(UniValue::VOBJ);
|
||||
result.push_back(Pair("txid", "txid-" + std::to_string(tag)));
|
||||
op->ForceSuccess(result);
|
||||
} else {
|
||||
op->ForceSuccess(UniValue("done"));
|
||||
}
|
||||
break;
|
||||
case FAKE_FAILED:
|
||||
op->ForceFailed(-6, "boom " + std::to_string(tag));
|
||||
break;
|
||||
case FAKE_CANCELLED:
|
||||
op->cancel(); // READY -> CANCELLED, the same path shutdown and wallet.cpp use
|
||||
break;
|
||||
}
|
||||
return op;
|
||||
}
|
||||
|
||||
bool IsFinished(const AsyncRPCOperation& op)
|
||||
{
|
||||
return op.isSuccess() || op.isFailed() || op.isCancelled();
|
||||
}
|
||||
|
||||
// The z_getoperationstatus_IMPL of c1040028e, verbatim except that the LOCK2 and the
|
||||
// popOperationForId call are left out. Evaluated on an unchanged queue just before the call under
|
||||
// test, it sees the same getAllOperationIds() enumeration and returns what the original returned:
|
||||
// the original removing pass popped each operation only after reading it, so it reported exactly
|
||||
// what this non-removing pass reports.
|
||||
UniValue ReferenceOperationStatus(const UniValue& params, bool fRemoveFinishedOperations)
|
||||
{
|
||||
std::set<AsyncRPCOperationId> filter;
|
||||
if (params.size()==1) {
|
||||
UniValue ids = params[0].get_array();
|
||||
for (const UniValue & v : ids.getValues()) {
|
||||
filter.insert(v.get_str());
|
||||
}
|
||||
}
|
||||
bool useFilter = (filter.size()>0);
|
||||
|
||||
UniValue ret(UniValue::VARR);
|
||||
std::shared_ptr<AsyncRPCQueue> q = getAsyncRPCQueue();
|
||||
std::vector<AsyncRPCOperationId> ids = q->getAllOperationIds();
|
||||
|
||||
for (auto id : ids) {
|
||||
if (useFilter && !filter.count(id))
|
||||
continue;
|
||||
|
||||
std::shared_ptr<AsyncRPCOperation> operation = q->getOperationForId(id);
|
||||
if (!operation) {
|
||||
continue;
|
||||
}
|
||||
|
||||
UniValue obj = operation->getStatus();
|
||||
std::string s = obj["status"].get_str();
|
||||
if (fRemoveFinishedOperations) {
|
||||
// Caller is only interested in retrieving finished results
|
||||
if ("success"==s || "failed"==s || "cancelled"==s) {
|
||||
ret.push_back(obj);
|
||||
}
|
||||
} else {
|
||||
ret.push_back(obj);
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<UniValue> arrTmp = ret.getValues();
|
||||
|
||||
// sort results chronologically by creation_time
|
||||
std::sort(arrTmp.begin(), arrTmp.end(), [](UniValue a, UniValue b) -> bool {
|
||||
const int64_t t1 = find_value(a.get_obj(), "creation_time").get_int64();
|
||||
const int64_t t2 = find_value(b.get_obj(), "creation_time").get_int64();
|
||||
return t1 < t2;
|
||||
});
|
||||
|
||||
ret.clear();
|
||||
ret.setArray();
|
||||
ret.push_backV(arrTmp);
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
// The body of c1040028e's z_listoperationids after its argument check, verbatim without the LOCK2.
|
||||
UniValue ReferenceListOperationIds(const UniValue& params)
|
||||
{
|
||||
std::string filter;
|
||||
bool useFilter = false;
|
||||
if (params.size()==1) {
|
||||
filter = params[0].get_str();
|
||||
useFilter = true;
|
||||
}
|
||||
|
||||
UniValue ret(UniValue::VARR);
|
||||
std::shared_ptr<AsyncRPCQueue> q = getAsyncRPCQueue();
|
||||
std::vector<AsyncRPCOperationId> ids = q->getAllOperationIds();
|
||||
for (auto id : ids) {
|
||||
std::shared_ptr<AsyncRPCOperation> operation = q->getOperationForId(id);
|
||||
if (!operation) {
|
||||
continue;
|
||||
}
|
||||
std::string state = operation->getStateAsString();
|
||||
if (useFilter && filter.compare(state)!=0)
|
||||
continue;
|
||||
ret.push_back(id);
|
||||
}
|
||||
|
||||
return ret;
|
||||
}
|
||||
|
||||
// Results and errors as one comparable string. These helpers make no gtest calls, so helper
|
||||
// threads can use them.
|
||||
std::string CallOutcome(rpcfn_type fn, const UniValue& params)
|
||||
{
|
||||
try {
|
||||
return fn(params, false, CPubKey()).write();
|
||||
} catch (const UniValue& objError) {
|
||||
return "rpc error: " + objError.write();
|
||||
} catch (const std::exception& e) {
|
||||
return std::string("exception: ") + e.what();
|
||||
}
|
||||
}
|
||||
|
||||
std::string ReferenceOutcome(const UniValue& params, bool fRemoveFinishedOperations)
|
||||
{
|
||||
try {
|
||||
return ReferenceOperationStatus(params, fRemoveFinishedOperations).write();
|
||||
} catch (const UniValue& objError) {
|
||||
return "rpc error: " + objError.write();
|
||||
} catch (const std::exception& e) {
|
||||
return std::string("exception: ") + e.what();
|
||||
}
|
||||
}
|
||||
|
||||
std::string ReferenceListOutcome(const UniValue& params)
|
||||
{
|
||||
try {
|
||||
return ReferenceListOperationIds(params).write();
|
||||
} catch (const UniValue& objError) {
|
||||
return "rpc error: " + objError.write();
|
||||
} catch (const std::exception& e) {
|
||||
return std::string("exception: ") + e.what();
|
||||
}
|
||||
}
|
||||
|
||||
// Byte comparison that reports where two (possibly very long) JSON texts first differ.
|
||||
::testing::AssertionResult SameText(const std::string& original, const std::string& patched)
|
||||
{
|
||||
if (original == patched)
|
||||
return ::testing::AssertionSuccess();
|
||||
size_t i = 0;
|
||||
while (i < original.size() && i < patched.size() && original[i] == patched[i])
|
||||
i++;
|
||||
const size_t from = i < 40 ? 0 : i - 40;
|
||||
return ::testing::AssertionFailure()
|
||||
<< "outputs differ at byte " << i << " (lengths " << original.size() << " / " << patched.size() << ")"
|
||||
<< "\n original: ..." << original.substr(from, 120)
|
||||
<< "\n patched: ..." << patched.substr(from, 120);
|
||||
}
|
||||
|
||||
UniValue NoParams()
|
||||
{
|
||||
return UniValue(UniValue::VARR);
|
||||
}
|
||||
|
||||
// [[id, ...]]
|
||||
UniValue IdParams(const std::vector<std::string>& ids)
|
||||
{
|
||||
UniValue arr(UniValue::VARR);
|
||||
for (size_t i = 0; i < ids.size(); i++)
|
||||
arr.push_back(ids[i]);
|
||||
UniValue params(UniValue::VARR);
|
||||
params.push_back(arr);
|
||||
return params;
|
||||
}
|
||||
|
||||
// The ids a params array asks for. Empty means "all", as in the handlers.
|
||||
std::set<std::string> RequestedIds(const UniValue& params)
|
||||
{
|
||||
std::set<std::string> ids;
|
||||
if (params.size() == 1) {
|
||||
const std::vector<UniValue>& v = params[0].get_array().getValues();
|
||||
for (size_t i = 0; i < v.size(); i++)
|
||||
ids.insert(v[i].get_str());
|
||||
}
|
||||
return ids;
|
||||
}
|
||||
|
||||
std::set<std::string> ListedIds()
|
||||
{
|
||||
UniValue listed = z_listoperationids(NoParams(), false, CPubKey());
|
||||
std::set<std::string> ids;
|
||||
for (size_t i = 0; i < listed.size(); i++)
|
||||
ids.insert(listed[i].get_str());
|
||||
return ids;
|
||||
}
|
||||
|
||||
// Installs an empty in-memory wallet as pwalletMain for one test, and when it goes out of scope
|
||||
// removes every operation the test added to the shared queue and restores pwalletMain.
|
||||
class TestEnv {
|
||||
public:
|
||||
TestEnv() : queue_(getAsyncRPCQueue()), savedWallet_(pwalletMain) {
|
||||
wasEmpty_ = queue_->getAllOperationIds().empty();
|
||||
pwalletMain = &wallet_;
|
||||
}
|
||||
|
||||
~TestEnv() {
|
||||
RemoveAll();
|
||||
pwalletMain = savedWallet_;
|
||||
}
|
||||
|
||||
bool QueueWasEmpty() const { return wasEmpty_; }
|
||||
|
||||
const std::shared_ptr<AsyncRPCQueue>& Queue() const { return queue_; }
|
||||
|
||||
// False if the queue refused an operation.
|
||||
bool AddAll(const std::vector<std::shared_ptr<FixedStateOperation> >& ops) {
|
||||
for (size_t i = 0; i < ops.size(); i++) {
|
||||
if (!queue_->addOperation(ops[i]))
|
||||
return false;
|
||||
added_.push_back(ops[i]->getId());
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
void RemoveAll() {
|
||||
for (size_t i = 0; i < added_.size(); i++)
|
||||
queue_->popOperationForId(added_[i]);
|
||||
added_.clear();
|
||||
}
|
||||
|
||||
private:
|
||||
std::shared_ptr<AsyncRPCQueue> queue_;
|
||||
CWallet wallet_;
|
||||
CWallet* savedWallet_;
|
||||
bool wasEmpty_;
|
||||
std::vector<AsyncRPCOperationId> added_;
|
||||
};
|
||||
|
||||
// Holds cs_main and pwalletMain->cs_wallet on a helper thread, as a long wallet scan does, until
|
||||
// Release() or destruction.
|
||||
class WalletLockHolder {
|
||||
public:
|
||||
WalletLockHolder() : held_(false), release_(false), thread_(&WalletLockHolder::Run, this) {}
|
||||
|
||||
~WalletLockHolder() {
|
||||
Release();
|
||||
thread_.join();
|
||||
}
|
||||
|
||||
bool WaitUntilHeld(std::chrono::milliseconds timeout) {
|
||||
std::unique_lock<std::mutex> lk(m_);
|
||||
return cv_.wait_for(lk, timeout, [this]() { return held_; });
|
||||
}
|
||||
|
||||
void Release() {
|
||||
{
|
||||
std::lock_guard<std::mutex> guard(m_);
|
||||
release_ = true;
|
||||
}
|
||||
cv_.notify_all();
|
||||
}
|
||||
|
||||
private:
|
||||
void Run() {
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
std::unique_lock<std::mutex> lk(m_);
|
||||
held_ = true;
|
||||
cv_.notify_all();
|
||||
cv_.wait(lk, [this]() { return release_; });
|
||||
}
|
||||
|
||||
std::mutex m_;
|
||||
std::condition_variable cv_;
|
||||
bool held_;
|
||||
bool release_;
|
||||
std::thread thread_; // last, so that everything Run() touches exists before it starts
|
||||
};
|
||||
|
||||
// creation_time is time(NULL) at construction and cannot be set, so ties and distinct times are
|
||||
// made by creating operations inside chosen seconds. Waits in 10 ms steps for the next second.
|
||||
void WaitForNextSecond()
|
||||
{
|
||||
const time_t start = time(NULL);
|
||||
for (int i = 0; i < 150 && time(NULL) == start; i++)
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds(10));
|
||||
}
|
||||
|
||||
bool AllCreatedInOneSecond(const std::vector<std::shared_ptr<FixedStateOperation> >& ops)
|
||||
{
|
||||
for (size_t i = 1; i < ops.size(); i++) {
|
||||
if (ops[i]->getCreationTime() != ops[0]->getCreationTime())
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
const int kPoolGroups = 3;
|
||||
const size_t kPoolGroupSize = 300;
|
||||
|
||||
// kPoolGroups groups of operations in all five states. Every operation in a group has the same
|
||||
// creation_time, and the groups' creation_times strictly increase.
|
||||
struct OperationPool {
|
||||
bool ok;
|
||||
std::vector<std::vector<std::shared_ptr<FixedStateOperation> > > groups;
|
||||
};
|
||||
|
||||
OperationPool BuildPool()
|
||||
{
|
||||
OperationPool pool;
|
||||
pool.ok = true;
|
||||
for (int g = 0; g < kPoolGroups && pool.ok; g++) {
|
||||
std::vector<std::shared_ptr<FixedStateOperation> > group;
|
||||
bool tied = false;
|
||||
for (int attempt = 0; attempt < 3 && !tied; attempt++) {
|
||||
if (g > 0 || attempt > 0)
|
||||
WaitForNextSecond();
|
||||
group.clear();
|
||||
for (size_t i = 0; i < kPoolGroupSize; i++) {
|
||||
group.push_back(MakeOperation(static_cast<int>((i + 2 * g) % FAKE_KINDS),
|
||||
static_cast<int>(g * kPoolGroupSize + i)));
|
||||
}
|
||||
tied = AllCreatedInOneSecond(group);
|
||||
}
|
||||
const bool later = pool.groups.empty() ||
|
||||
group.front()->getCreationTime() > pool.groups.back().front()->getCreationTime();
|
||||
pool.ok = tied && later;
|
||||
pool.groups.push_back(group);
|
||||
}
|
||||
return pool;
|
||||
}
|
||||
|
||||
// Built once (it waits for up to about two second boundaries) and shared by the tests that need
|
||||
// it. The operations are never modified after they are built.
|
||||
const OperationPool& SharedPool()
|
||||
{
|
||||
static const OperationPool pool = BuildPool();
|
||||
return pool;
|
||||
}
|
||||
|
||||
struct Population {
|
||||
std::string name;
|
||||
std::vector<std::shared_ptr<FixedStateOperation> > ops;
|
||||
};
|
||||
|
||||
// For each size: "ties-K", K operations from one second, and "mixed-K", K operations spread
|
||||
// round robin over three seconds (ties within each second). 16 is the largest size that
|
||||
// libstdc++'s std::sort handles with insertion sort alone; from 17 on, introsort reorders equal keys.
|
||||
std::vector<Population> Populations(const OperationPool& pool)
|
||||
{
|
||||
static const size_t sizes[] = {0, 1, 2, 16, 17, 40, 300};
|
||||
std::vector<Population> pops;
|
||||
for (size_t s = 0; s < sizeof(sizes) / sizeof(sizes[0]); s++) {
|
||||
const size_t k = sizes[s];
|
||||
|
||||
Population ties;
|
||||
ties.name = "ties-" + std::to_string(k);
|
||||
ties.ops.assign(pool.groups[1].begin(), pool.groups[1].begin() + k);
|
||||
pops.push_back(ties);
|
||||
|
||||
Population mixed;
|
||||
mixed.name = "mixed-" + std::to_string(k);
|
||||
for (size_t i = 0; i < k; i++)
|
||||
mixed.ops.push_back(pool.groups[i % kPoolGroups][i / kPoolGroups]);
|
||||
pops.push_back(mixed);
|
||||
}
|
||||
return pops;
|
||||
}
|
||||
|
||||
// Arguments for z_getoperationstatus/z_getoperationresult over a population: no argument, an
|
||||
// empty list (no filter), an unknown id, two ids, a duplicated id, every other id plus an
|
||||
// unknown one, and every id.
|
||||
std::vector<UniValue> StatusParamSets(const std::vector<std::shared_ptr<FixedStateOperation> >& ops)
|
||||
{
|
||||
std::vector<UniValue> sets;
|
||||
sets.push_back(NoParams());
|
||||
sets.push_back(IdParams(std::vector<std::string>()));
|
||||
sets.push_back(IdParams(std::vector<std::string>(1, "opid-unknown")));
|
||||
if (!ops.empty()) {
|
||||
std::vector<std::string> two;
|
||||
two.push_back(ops.front()->getId());
|
||||
two.push_back(ops.back()->getId());
|
||||
sets.push_back(IdParams(two));
|
||||
sets.push_back(IdParams(std::vector<std::string>(2, ops.front()->getId())));
|
||||
|
||||
std::vector<std::string> half, all;
|
||||
for (size_t i = 0; i < ops.size(); i++) {
|
||||
all.push_back(ops[i]->getId());
|
||||
if (i % 2 == 0)
|
||||
half.push_back(ops[i]->getId());
|
||||
}
|
||||
half.push_back("opid-unknown");
|
||||
sets.push_back(IdParams(half));
|
||||
sets.push_back(IdParams(all));
|
||||
}
|
||||
return sets;
|
||||
}
|
||||
|
||||
// Pops every id once, as fast as it can, after the start signal.
|
||||
void PopEach(AsyncRPCQueue* q, const std::vector<AsyncRPCOperationId>* ids,
|
||||
const std::atomic<bool>* go, std::vector<AsyncRPCOperationId>* out)
|
||||
{
|
||||
while (!go->load())
|
||||
std::this_thread::yield();
|
||||
for (size_t i = 0; i < ids->size(); i++) {
|
||||
if (q->popOperationForId((*ids)[i]) != nullptr)
|
||||
out->push_back((*ids)[i]);
|
||||
}
|
||||
}
|
||||
|
||||
// Removes finished operations until the queue is empty (or kGuardTimeout passes) and records
|
||||
// every id it was handed. If direct is true it calls the IMPL without cs_main/cs_wallet, so only
|
||||
// the queue's atomic pop and the IMPL's checked pop keep it apart from the other removers.
|
||||
// Otherwise it calls the z_getoperationresult handler, which takes the locks.
|
||||
void DrainFinished(bool direct, const std::atomic<bool>* go, std::vector<std::string>* out)
|
||||
{
|
||||
while (!go->load())
|
||||
std::this_thread::yield();
|
||||
const UniValue params(UniValue::VARR);
|
||||
std::shared_ptr<AsyncRPCQueue> q = getAsyncRPCQueue();
|
||||
const std::chrono::steady_clock::time_point deadline = std::chrono::steady_clock::now() + kGuardTimeout;
|
||||
while (std::chrono::steady_clock::now() < deadline) {
|
||||
UniValue ret;
|
||||
try {
|
||||
ret = direct ? z_getoperationstatus_IMPL(params, true)
|
||||
: z_getoperationresult(params, false, CPubKey());
|
||||
for (size_t i = 0; i < ret.size(); i++)
|
||||
out->push_back(find_value(ret[i], "id").get_str());
|
||||
} catch (...) {
|
||||
out->push_back("<exception>");
|
||||
return;
|
||||
}
|
||||
if (ret.empty() && q->getAllOperationIds().empty())
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// Polls z_getoperationstatus while the removers run. Counts results that break its contract
|
||||
// (only the test's finished operations exist): an unfinished status, a repeated id, or
|
||||
// creation_time out of order.
|
||||
void PollStatus(const std::atomic<bool>* go, const std::atomic<bool>* stop, int* faults)
|
||||
{
|
||||
while (!go->load())
|
||||
std::this_thread::yield();
|
||||
const UniValue params(UniValue::VARR);
|
||||
while (!stop->load()) {
|
||||
try {
|
||||
UniValue ret = z_getoperationstatus(params, false, CPubKey());
|
||||
std::set<std::string> ids;
|
||||
int64_t last = 0;
|
||||
for (size_t i = 0; i < ret.size(); i++) {
|
||||
const std::string id = find_value(ret[i], "id").get_str();
|
||||
const std::string status = find_value(ret[i], "status").get_str();
|
||||
const int64_t t = find_value(ret[i], "creation_time").get_int64();
|
||||
if (!ids.insert(id).second || t < last ||
|
||||
(status != "success" && status != "failed" && status != "cancelled"))
|
||||
(*faults)++;
|
||||
last = t;
|
||||
}
|
||||
} catch (...) {
|
||||
(*faults)++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// The read path returns exactly what the original algorithm returned, including the order of
|
||||
// same-second operations, and removes nothing.
|
||||
TEST(OperationStatus, StatusMatchesOriginalAlgorithm)
|
||||
{
|
||||
const OperationPool& pool = SharedPool();
|
||||
ASSERT_TRUE(pool.ok) << kPoolFailed;
|
||||
TestEnv env;
|
||||
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
|
||||
|
||||
const std::vector<Population> pops = Populations(pool);
|
||||
for (size_t p = 0; p < pops.size(); p++) {
|
||||
ASSERT_TRUE(env.AddAll(pops[p].ops));
|
||||
const std::vector<UniValue> paramSets = StatusParamSets(pops[p].ops);
|
||||
for (size_t i = 0; i < paramSets.size(); i++) {
|
||||
EXPECT_TRUE(SameText(ReferenceOutcome(paramSets[i], false),
|
||||
CallOutcome(&z_getoperationstatus, paramSets[i])))
|
||||
<< pops[p].name << " params=" << paramSets[i].write();
|
||||
}
|
||||
EXPECT_EQ(pops[p].ops.size(), env.Queue()->getAllOperationIds().size()) << pops[p].name;
|
||||
env.RemoveAll();
|
||||
}
|
||||
|
||||
#if defined(__GLIBCXX__)
|
||||
// Check that the comparison above can fail. With 40 operations in one second, libstdc++'s
|
||||
// introsort reorders the equal keys, so the original output is not the enumeration order.
|
||||
// A (creation_time, index) comparator would keep the enumeration order and fail above.
|
||||
std::vector<std::shared_ptr<FixedStateOperation> > tied(pool.groups[1].begin(), pool.groups[1].begin() + 40);
|
||||
ASSERT_TRUE(env.AddAll(tied));
|
||||
const UniValue ref = ReferenceOperationStatus(NoParams(), false);
|
||||
std::vector<std::string> reported;
|
||||
for (size_t i = 0; i < ref.size(); i++)
|
||||
reported.push_back(find_value(ref[i], "id").get_str());
|
||||
const std::vector<std::string> enumerated = env.Queue()->getAllOperationIds();
|
||||
EXPECT_EQ(enumerated.size(), reported.size());
|
||||
EXPECT_NE(enumerated, reported) << "40 tied operations kept their enumeration order";
|
||||
env.RemoveAll();
|
||||
#endif
|
||||
}
|
||||
|
||||
// The removing path reports exactly what the original reported, removes exactly the finished
|
||||
// operations it was asked about, and leaves the rest visible to the other two calls.
|
||||
TEST(OperationStatus, ResultMatchesOriginalAlgorithm)
|
||||
{
|
||||
const OperationPool& pool = SharedPool();
|
||||
ASSERT_TRUE(pool.ok) << kPoolFailed;
|
||||
TestEnv env;
|
||||
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
|
||||
|
||||
const std::vector<Population> pops = Populations(pool);
|
||||
for (size_t p = 0; p < pops.size(); p++) {
|
||||
const std::vector<UniValue> paramSets = StatusParamSets(pops[p].ops);
|
||||
for (size_t i = 0; i < paramSets.size(); i++) {
|
||||
const UniValue& params = paramSets[i];
|
||||
const std::string where = pops[p].name + " params=" + params.write();
|
||||
ASSERT_TRUE(env.AddAll(pops[p].ops));
|
||||
|
||||
const std::string ref = ReferenceOutcome(params, true);
|
||||
EXPECT_TRUE(SameText(ref, CallOutcome(&z_getoperationresult, params))) << where;
|
||||
|
||||
const std::set<std::string> requested = RequestedIds(params);
|
||||
std::set<std::string> remaining;
|
||||
int wrong = 0;
|
||||
for (size_t j = 0; j < pops[p].ops.size(); j++) {
|
||||
const std::shared_ptr<FixedStateOperation>& op = pops[p].ops[j];
|
||||
const bool selected = requested.empty() || requested.count(op->getId()) > 0;
|
||||
const bool present = env.Queue()->getOperationForId(op->getId()) != nullptr;
|
||||
if (present != !(IsFinished(*op) && selected))
|
||||
wrong++;
|
||||
if (present)
|
||||
remaining.insert(op->getId());
|
||||
}
|
||||
EXPECT_EQ(0, wrong) << where << ": operations removed or kept wrongly";
|
||||
|
||||
// Nothing is handed out twice, and the survivors are still listed and reported.
|
||||
EXPECT_EQ("[]", CallOutcome(&z_getoperationresult, params)) << where;
|
||||
EXPECT_EQ(remaining, ListedIds()) << where;
|
||||
EXPECT_TRUE(SameText(ReferenceOutcome(NoParams(), false),
|
||||
CallOutcome(&z_getoperationstatus, NoParams()))) << where;
|
||||
|
||||
env.RemoveAll();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
TEST(OperationStatus, ListOperationIdsMatchesOriginalAlgorithm)
|
||||
{
|
||||
const OperationPool& pool = SharedPool();
|
||||
ASSERT_TRUE(pool.ok) << kPoolFailed;
|
||||
TestEnv env;
|
||||
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
|
||||
|
||||
const char* const filters[] = {NULL, "queued", "executing", "success", "failed", "cancelled", "bogus"};
|
||||
const std::vector<Population> pops = Populations(pool);
|
||||
for (size_t p = 0; p < pops.size(); p++) {
|
||||
ASSERT_TRUE(env.AddAll(pops[p].ops));
|
||||
for (size_t f = 0; f < sizeof(filters) / sizeof(filters[0]); f++) {
|
||||
UniValue params(UniValue::VARR);
|
||||
if (filters[f] != NULL)
|
||||
params.push_back(filters[f]);
|
||||
EXPECT_TRUE(SameText(ReferenceListOutcome(params), CallOutcome(&z_listoperationids, params)))
|
||||
<< pops[p].name << " params=" << params.write();
|
||||
}
|
||||
env.RemoveAll();
|
||||
}
|
||||
}
|
||||
|
||||
// Malformed arguments fail exactly as before and remove nothing; no wallet still means
|
||||
// "Method not found".
|
||||
TEST(OperationStatus, ParameterErrorsUnchanged)
|
||||
{
|
||||
TestEnv env;
|
||||
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
|
||||
std::vector<std::shared_ptr<FixedStateOperation> > ops;
|
||||
for (int k = 0; k < FAKE_KINDS; k++)
|
||||
ops.push_back(MakeOperation(k, k));
|
||||
ASSERT_TRUE(env.AddAll(ops));
|
||||
|
||||
UniValue notArray(UniValue::VARR);
|
||||
notArray.push_back("not_an_array");
|
||||
UniValue inner(UniValue::VARR);
|
||||
inner.push_back(ops[FAKE_SUCCESS]->getId());
|
||||
inner.push_back(1);
|
||||
UniValue notString(UniValue::VARR);
|
||||
notString.push_back(inner);
|
||||
|
||||
EXPECT_EQ("exception: JSON value is not an array as expected", ReferenceOutcome(notArray, false));
|
||||
EXPECT_EQ("exception: JSON value is not a string as expected", ReferenceOutcome(notString, false));
|
||||
EXPECT_EQ(ReferenceOutcome(notArray, false), CallOutcome(&z_getoperationstatus, notArray));
|
||||
EXPECT_EQ(ReferenceOutcome(notString, false), CallOutcome(&z_getoperationstatus, notString));
|
||||
EXPECT_EQ(ReferenceOutcome(notArray, true), CallOutcome(&z_getoperationresult, notArray));
|
||||
EXPECT_EQ(ReferenceOutcome(notString, true), CallOutcome(&z_getoperationresult, notString));
|
||||
|
||||
UniValue tooMany(UniValue::VARR);
|
||||
tooMany.push_back(UniValue(UniValue::VARR));
|
||||
tooMany.push_back("toomanyargs");
|
||||
EXPECT_THROW(z_getoperationstatus(tooMany, false, CPubKey()), std::runtime_error);
|
||||
EXPECT_THROW(z_getoperationresult(tooMany, false, CPubKey()), std::runtime_error);
|
||||
EXPECT_THROW(z_listoperationids(tooMany, false, CPubKey()), std::runtime_error);
|
||||
|
||||
EXPECT_EQ(ops.size(), env.Queue()->getAllOperationIds().size());
|
||||
|
||||
CWallet* wallet = pwalletMain;
|
||||
pwalletMain = NULL;
|
||||
EXPECT_THROW(z_getoperationstatus(NoParams(), false, CPubKey()), UniValue);
|
||||
EXPECT_THROW(z_getoperationresult(NoParams(), false, CPubKey()), UniValue);
|
||||
EXPECT_THROW(z_listoperationids(NoParams(), false, CPubKey()), UniValue);
|
||||
pwalletMain = wallet;
|
||||
}
|
||||
|
||||
// While another thread holds cs_main + cs_wallet, z_getoperationstatus (unfiltered and filtered,
|
||||
// as the GUI polls it) and z_listoperationids still answer, with unchanged output, and
|
||||
// z_getoperationresult waits for the locks and then answers as before.
|
||||
TEST(OperationStatus, StatusDoesNotWaitForWalletLocks)
|
||||
{
|
||||
TestEnv env;
|
||||
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
|
||||
std::vector<std::shared_ptr<FixedStateOperation> > ops;
|
||||
for (int k = 0; k < FAKE_KINDS; k++)
|
||||
ops.push_back(MakeOperation(k, k));
|
||||
ASSERT_TRUE(env.AddAll(ops));
|
||||
|
||||
const UniValue filtered = IdParams(std::vector<std::string>(1, ops[FAKE_SUCCESS]->getId()));
|
||||
const std::string refStatus = ReferenceOutcome(NoParams(), false);
|
||||
const std::string refFiltered = ReferenceOutcome(filtered, false);
|
||||
const std::string refList = ReferenceListOutcome(NoParams());
|
||||
const std::string refResult = ReferenceOutcome(NoParams(), true);
|
||||
|
||||
// Declared before the holder: on an early return the holder is destroyed first, which
|
||||
// releases the locks, and only then do the futures' destructors wait for their calls.
|
||||
std::future<std::string> status, statusFiltered, listed, result;
|
||||
WalletLockHolder holder;
|
||||
ASSERT_TRUE(holder.WaitUntilHeld(kGuardTimeout)) << "helper thread could not take cs_main + cs_wallet";
|
||||
|
||||
status = std::async(std::launch::async, CallOutcome, &z_getoperationstatus, NoParams());
|
||||
statusFiltered = std::async(std::launch::async, CallOutcome, &z_getoperationstatus, filtered);
|
||||
listed = std::async(std::launch::async, CallOutcome, &z_listoperationids, NoParams());
|
||||
result = std::async(std::launch::async, CallOutcome, &z_getoperationresult, NoParams());
|
||||
|
||||
const bool statusDone = status.wait_for(kGuardTimeout) == std::future_status::ready;
|
||||
const bool filteredDone = statusFiltered.wait_for(kGuardTimeout) == std::future_status::ready;
|
||||
const bool listDone = listed.wait_for(kGuardTimeout) == std::future_status::ready;
|
||||
// The locks are still held, so a z_getoperationresult that takes them cannot have finished.
|
||||
const bool resultWaited = result.wait_for(std::chrono::milliseconds(100)) == std::future_status::timeout;
|
||||
holder.Release();
|
||||
const bool resultDone = result.wait_for(kGuardTimeout) == std::future_status::ready;
|
||||
|
||||
EXPECT_TRUE(statusDone) << "z_getoperationstatus waited for cs_main/cs_wallet";
|
||||
EXPECT_TRUE(filteredDone) << "filtered z_getoperationstatus waited for cs_main/cs_wallet";
|
||||
EXPECT_TRUE(listDone) << "z_listoperationids waited for cs_main/cs_wallet";
|
||||
EXPECT_TRUE(resultWaited) << "z_getoperationresult no longer waits for cs_main/cs_wallet";
|
||||
EXPECT_TRUE(resultDone) << "z_getoperationresult did not finish after the locks were released";
|
||||
if (statusDone) {
|
||||
EXPECT_EQ(refStatus, status.get());
|
||||
}
|
||||
if (filteredDone) {
|
||||
EXPECT_EQ(refFiltered, statusFiltered.get());
|
||||
}
|
||||
if (listDone) {
|
||||
EXPECT_EQ(refList, listed.get());
|
||||
}
|
||||
if (resultDone) {
|
||||
EXPECT_EQ(refResult, result.get());
|
||||
}
|
||||
|
||||
// The result call removed the three finished operations and kept the queued and executing ones.
|
||||
std::set<std::string> expected;
|
||||
expected.insert(ops[FAKE_QUEUED]->getId());
|
||||
expected.insert(ops[FAKE_EXECUTING]->getId());
|
||||
EXPECT_EQ(expected, ListedIds());
|
||||
}
|
||||
|
||||
// SC-1: popOperationForId looks up and erases under one hold of the queue lock, so when several
|
||||
// threads pop the same ids each operation goes to exactly one of them. The old two-step pop
|
||||
// could hand the same operation to two callers; this catches that probabilistically.
|
||||
TEST(OperationStatus, PopOperationForIdHandsEachOperationToOneCaller)
|
||||
{
|
||||
{
|
||||
std::shared_ptr<AsyncRPCQueue> q = std::make_shared<AsyncRPCQueue>();
|
||||
std::shared_ptr<FixedStateOperation> op = MakeOperation(FAKE_SUCCESS, 1);
|
||||
ASSERT_TRUE(q->addOperation(op));
|
||||
EXPECT_TRUE(q->popOperationForId("opid-unknown") == nullptr);
|
||||
EXPECT_TRUE(q->popOperationForId(op->getId()).get() == op.get());
|
||||
EXPECT_TRUE(q->getOperationForId(op->getId()) == nullptr);
|
||||
EXPECT_TRUE(q->popOperationForId(op->getId()) == nullptr);
|
||||
}
|
||||
|
||||
const int kThreads = 8;
|
||||
const int kOps = 1000;
|
||||
const int kRounds = 10;
|
||||
for (int round = 0; round < kRounds; round++) {
|
||||
// A private queue with no workers.
|
||||
std::shared_ptr<AsyncRPCQueue> q = std::make_shared<AsyncRPCQueue>();
|
||||
std::vector<AsyncRPCOperationId> ids;
|
||||
for (int i = 0; i < kOps; i++) {
|
||||
std::shared_ptr<AsyncRPCOperation> op(new FixedStateOperation());
|
||||
ASSERT_TRUE(q->addOperation(op));
|
||||
ids.push_back(op->getId());
|
||||
}
|
||||
|
||||
std::atomic<bool> go(false);
|
||||
std::vector<std::vector<AsyncRPCOperationId> > popped(kThreads);
|
||||
std::vector<std::thread> threads;
|
||||
for (int t = 0; t < kThreads; t++)
|
||||
threads.push_back(std::thread(PopEach, q.get(), &ids, &go, &popped[t]));
|
||||
go.store(true);
|
||||
for (size_t t = 0; t < threads.size(); t++)
|
||||
threads[t].join();
|
||||
|
||||
std::map<AsyncRPCOperationId, int> handed;
|
||||
size_t total = 0;
|
||||
for (int t = 0; t < kThreads; t++) {
|
||||
for (size_t i = 0; i < popped[t].size(); i++) {
|
||||
handed[popped[t][i]]++;
|
||||
total++;
|
||||
}
|
||||
}
|
||||
int wrong = 0;
|
||||
for (size_t i = 0; i < ids.size(); i++) {
|
||||
std::map<AsyncRPCOperationId, int>::const_iterator it = handed.find(ids[i]);
|
||||
if (it == handed.end() || it->second != 1)
|
||||
wrong++;
|
||||
}
|
||||
EXPECT_EQ(0, wrong) << "round " << round << ": operations not handed to exactly one caller";
|
||||
EXPECT_EQ(static_cast<size_t>(kOps), total) << "round " << round;
|
||||
EXPECT_TRUE(q->getAllOperationIds().empty()) << "round " << round;
|
||||
}
|
||||
}
|
||||
|
||||
// Removers through the locked handler and straight through the IMPL (no wallet locks) drain the
|
||||
// same finished operations while two lock-free z_getoperationstatus pollers run. Every operation
|
||||
// is reported exactly once, and no poll returns a malformed or unsorted list.
|
||||
TEST(OperationStatus, ConcurrentRemoversReportEachOperationOnce)
|
||||
{
|
||||
TestEnv env;
|
||||
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
|
||||
const int kOps = 200;
|
||||
std::vector<std::shared_ptr<FixedStateOperation> > ops;
|
||||
std::set<std::string> expected;
|
||||
for (int i = 0; i < kOps; i++) {
|
||||
ops.push_back(MakeOperation(FAKE_SUCCESS + i % 3, i)); // success, failed or cancelled
|
||||
expected.insert(ops.back()->getId());
|
||||
}
|
||||
ASSERT_TRUE(env.AddAll(ops));
|
||||
|
||||
const int kRemovers = 8;
|
||||
const int kReaders = 2;
|
||||
std::atomic<bool> go(false);
|
||||
std::atomic<bool> stop(false);
|
||||
std::vector<std::vector<std::string> > removed(kRemovers);
|
||||
std::vector<int> readerFaults(kReaders, 0);
|
||||
std::vector<std::thread> removers, readers;
|
||||
for (int t = 0; t < kRemovers; t++)
|
||||
removers.push_back(std::thread(DrainFinished, t % 2 == 0, &go, &removed[t]));
|
||||
for (int r = 0; r < kReaders; r++)
|
||||
readers.push_back(std::thread(PollStatus, &go, &stop, &readerFaults[r]));
|
||||
go.store(true);
|
||||
for (size_t t = 0; t < removers.size(); t++)
|
||||
removers[t].join();
|
||||
stop.store(true);
|
||||
for (size_t r = 0; r < readers.size(); r++)
|
||||
readers[r].join();
|
||||
|
||||
std::map<std::string, int> reported;
|
||||
size_t total = 0;
|
||||
for (int t = 0; t < kRemovers; t++) {
|
||||
for (size_t i = 0; i < removed[t].size(); i++) {
|
||||
reported[removed[t][i]]++;
|
||||
total++;
|
||||
}
|
||||
}
|
||||
int duplicated = 0, unknown = 0;
|
||||
for (std::map<std::string, int>::const_iterator it = reported.begin(); it != reported.end(); ++it) {
|
||||
if (it->second != 1)
|
||||
duplicated++;
|
||||
if (!expected.count(it->first))
|
||||
unknown++;
|
||||
}
|
||||
EXPECT_EQ(0, duplicated) << "an operation was reported by more than one remover";
|
||||
EXPECT_EQ(0, unknown) << "a remover reported an unknown id or threw";
|
||||
EXPECT_EQ(static_cast<size_t>(kOps), reported.size());
|
||||
EXPECT_EQ(static_cast<size_t>(kOps), total);
|
||||
EXPECT_TRUE(env.Queue()->getAllOperationIds().empty());
|
||||
for (int r = 0; r < kReaders; r++)
|
||||
EXPECT_EQ(0, readerFaults[r]) << "reader " << r;
|
||||
}
|
||||
|
||||
// SC-4: the const status map with .at() names every state exactly as before.
|
||||
TEST(OperationStatus, StateNamesUnchanged)
|
||||
{
|
||||
const char* const names[FAKE_KINDS] = {"queued", "executing", "success", "failed", "cancelled"};
|
||||
for (int k = 0; k < FAKE_KINDS; k++) {
|
||||
std::shared_ptr<FixedStateOperation> op = MakeOperation(k, k + 1);
|
||||
EXPECT_EQ(std::string(names[k]), op->getStateAsString());
|
||||
const UniValue status = op->getStatus();
|
||||
EXPECT_EQ(std::string(names[k]), find_value(status, "status").get_str());
|
||||
}
|
||||
}
|
||||
@@ -1,58 +0,0 @@
|
||||
// Copyright (c) 2016-2024 The Hush developers
|
||||
// Distributed under the GPLv3 software license, see the accompanying
|
||||
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <boost/filesystem.hpp>
|
||||
|
||||
#include "wallet/db.h"
|
||||
|
||||
namespace {
|
||||
|
||||
// A fresh environment in a temp dir with one committed record in "wallet.dat", so database/ holds a log file.
|
||||
class WalletDbEnv : public ::testing::Test {
|
||||
protected:
|
||||
boost::filesystem::path dir;
|
||||
CDBEnv env;
|
||||
|
||||
void SetUp() override {
|
||||
dir = boost::filesystem::temp_directory_path() / boost::filesystem::unique_path("dbenv-%%%%%%%%");
|
||||
boost::filesystem::create_directories(dir);
|
||||
ASSERT_TRUE(env.Open(dir));
|
||||
Db db(env.dbenv, DB_CXX_NO_EXCEPTIONS);
|
||||
ASSERT_EQ(0, db.open(NULL, "wallet.dat", "main", DB_BTREE, DB_CREATE | DB_AUTO_COMMIT, 0));
|
||||
char k[] = "key", v[] = "value";
|
||||
Dbt key(k, sizeof k), value(v, sizeof v);
|
||||
ASSERT_EQ(0, db.put(NULL, &key, &value, 0));
|
||||
ASSERT_EQ(0, db.close(0));
|
||||
ASSERT_TRUE(boost::filesystem::exists(Log()));
|
||||
}
|
||||
void TearDown() override {
|
||||
env.Close();
|
||||
boost::system::error_code ec;
|
||||
boost::filesystem::remove_all(dir, ec);
|
||||
}
|
||||
boost::filesystem::path Log() const { return dir / "database" / "log.0000000001"; }
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
TEST_F(WalletDbEnv, CleanShutdownRemovesTheLogs) {
|
||||
env.Flush(true);
|
||||
EXPECT_FALSE(boost::filesystem::exists(dir / "database"));
|
||||
}
|
||||
|
||||
// BDB0060 PANIC (a page write failed, e.g. through an fd closed behind BDB's back): the logs may hold committed
|
||||
// changes wallet.dat never got, and the next start's DB_RECOVER needs them.
|
||||
TEST_F(WalletDbEnv, PanickedEnvironmentKeepsTheLogs) {
|
||||
ASSERT_EQ(0, env.dbenv->set_flags(DB_PANIC_ENVIRONMENT, 1));
|
||||
env.Flush(true);
|
||||
EXPECT_TRUE(boost::filesystem::exists(Log()));
|
||||
}
|
||||
|
||||
TEST_F(WalletDbEnv, FailedCheckpointKeepsTheLogs) {
|
||||
env.CheckpointLSN("no-such-file.dat"); // lsn_reset fails
|
||||
env.Flush(true);
|
||||
EXPECT_TRUE(boost::filesystem::exists(Log()));
|
||||
}
|
||||
@@ -1,816 +0,0 @@
|
||||
// Copyright (c) 2016-2024 The Hush developers
|
||||
// Distributed under the GPLv3 software license, see the accompanying
|
||||
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <deque>
|
||||
#include <utf8.h>
|
||||
|
||||
#include "chain.h"
|
||||
#include "chainparams.h"
|
||||
#include "coins.h"
|
||||
#include "init.h"
|
||||
#include "key_io.h"
|
||||
#include "main.h"
|
||||
#include "primitives/block.h"
|
||||
#include "primitives/transaction.h"
|
||||
#include "random.h"
|
||||
#include "rpc/server.h"
|
||||
#include "script/standard.h"
|
||||
#include "utilmoneystr.h"
|
||||
#include "wallet/wallet.h"
|
||||
#include "wallet/rpchushwallet.h"
|
||||
#include "zcash/Address.hpp"
|
||||
#include "zcash/Note.hpp"
|
||||
#include "zcash/NoteEncryption.hpp"
|
||||
#include "zcash/zip32.h"
|
||||
|
||||
using namespace std;
|
||||
using namespace libzcash;
|
||||
|
||||
// The history-RPC helpers (src/wallet/rpchushwallet.cpp) as of 69bb035e7, verbatim apart from the Legacy_
|
||||
// prefix and static linkage: the reference the optimized helpers must reproduce byte for byte.
|
||||
static void Legacy_zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) {
|
||||
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
|
||||
//Used to identify incomplete key sets
|
||||
int vinCount = 0;
|
||||
int vinSpendCount = 0;
|
||||
int shieldedSpendCount = 0;
|
||||
int shieldedSpendsSpentCount = 0;
|
||||
|
||||
//Check address
|
||||
bool isTAddress = false;
|
||||
bool isZsAddress = false;
|
||||
|
||||
CTxDestination tAddress = DecodeDestination(strAddress);
|
||||
auto zAddress = DecodePaymentAddress(strAddress);
|
||||
SaplingPaymentAddress zsAddress;
|
||||
|
||||
if (filterByAddress) {
|
||||
|
||||
if (IsValidDestination(tAddress))
|
||||
isTAddress = true;
|
||||
|
||||
if (IsValidPaymentAddress(zAddress)) {
|
||||
if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
|
||||
zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
|
||||
isZsAddress = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// Transparent Inputs belonging to the wallet
|
||||
UniValue tSpends(UniValue::VARR);
|
||||
if (isTAddress || !filterByAddress) {
|
||||
for (int i = 0; i < wtx.vin.size(); i++) {
|
||||
vinCount++;
|
||||
const CTxIn& txin = wtx.vin[i];
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
CTxDestination address;
|
||||
const CWalletTx* parent = pwalletMain->GetWalletTx(txin.prevout.hash);
|
||||
if (parent != NULL) {
|
||||
const CTxOut& parentOut = parent->vout[txin.prevout.n];
|
||||
ExtractDestination(parentOut.scriptPubKey, address);
|
||||
if(IsMine(*pwalletMain, address)){
|
||||
vinSpendCount++;
|
||||
totalSpends += CAmount(-parentOut.nValue);
|
||||
obj.push_back(Pair("address",EncodeDestination(address)));
|
||||
obj.push_back(Pair("scriptPubKey",HexStr(parentOut.scriptPubKey.begin(), parentOut.scriptPubKey.end())));
|
||||
obj.push_back(Pair("amount",ValueFromAmount(-parentOut.nValue)));
|
||||
obj.push_back(Pair("spendTxid",parent->GetHash().ToString()));
|
||||
obj.push_back(Pair("spendVout",(int)txin.prevout.n));
|
||||
CTxDestination filterAddress = DecodeDestination(strAddress);
|
||||
if (address == tAddress || !filterByAddress) {
|
||||
filteredSpends += CAmount(-parentOut.nValue);
|
||||
tSpends.push_back(obj);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
spends.push_back(Pair("transparentSpends",tSpends));
|
||||
|
||||
// Sapling Inputs belonging to the wallet
|
||||
UniValue zsSpends(UniValue::VARR);
|
||||
if (isZsAddress || !filterByAddress) {
|
||||
for (int i = 0; i < wtx.vShieldedSpend.size(); i++) {
|
||||
shieldedSpendCount++;
|
||||
const SpendDescription& spendDesc = wtx.vShieldedSpend[i];
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
|
||||
SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier];
|
||||
|
||||
if (pwalletMain->IsSaplingNullifierFromMe(spendDesc.nullifier)) {
|
||||
const CWalletTx* parent = pwalletMain->GetWalletTx(pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash);
|
||||
const OutputDescription& output = parent->vShieldedOutput[op.n];
|
||||
auto nd = pwalletMain->mapWallet[pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash].mapSaplingNoteData[op];
|
||||
auto pt = libzcash::SaplingNotePlaintext::decrypt(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
|
||||
|
||||
if (pt) {
|
||||
auto note = pt.get();
|
||||
auto pa = nd.ivk.address(note.d);
|
||||
auto address = pa.get();
|
||||
shieldedSpendsSpentCount++;
|
||||
totalSpends += CAmount(-note.value());
|
||||
obj.push_back(Pair("address",EncodePaymentAddress(address)));
|
||||
obj.push_back(Pair("amount", ValueFromAmount(CAmount(-note.value()))));
|
||||
obj.push_back(Pair("spendTxid",parent->GetHash().ToString()));
|
||||
obj.push_back(Pair("spendshieldedOutputIndex",(int)op.n));
|
||||
if (address == zsAddress || !filterByAddress) {
|
||||
filteredSpends += CAmount(-note.value());
|
||||
zsSpends.push_back(obj);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
spends.push_back(Pair("saplingSpends",zsSpends));
|
||||
|
||||
|
||||
spends.push_back(Pair("totalSpends",ValueFromAmount(filteredSpends)));
|
||||
|
||||
if (!filterByAddress) {
|
||||
if (filteredSpends != 0 && (vinCount != vinSpendCount || shieldedSpendCount != shieldedSpendsSpentCount)) {
|
||||
spends.push_back(Pair("missingSpendingKeys", true));
|
||||
spends.push_back(Pair("vinCount", vinCount));
|
||||
spends.push_back(Pair("vinSpendCount", vinSpendCount));
|
||||
spends.push_back(Pair("shieldedSpendCount", shieldedSpendCount));
|
||||
spends.push_back(Pair("shieldedSpendsSpentCount", shieldedSpendsSpentCount));
|
||||
} else {
|
||||
spends.push_back(Pair("missingSpendingKeys", false));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
static void Legacy_zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress) {
|
||||
|
||||
if(fZdebug)
|
||||
fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
|
||||
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
|
||||
//Check address
|
||||
bool isTAddress = false;
|
||||
bool isZsAddress = false;
|
||||
|
||||
CTxDestination tAddress = DecodeDestination(strAddress);
|
||||
auto zAddress = DecodePaymentAddress(strAddress);
|
||||
SaplingPaymentAddress zsAddress;
|
||||
|
||||
if (filterByAddress) {
|
||||
|
||||
if (IsValidDestination(tAddress))
|
||||
isTAddress = true;
|
||||
|
||||
if (IsValidPaymentAddress(zAddress)) {
|
||||
if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
|
||||
zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
|
||||
isZsAddress = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
//Transparent Received txos belonging to the wallet
|
||||
UniValue tReceived(UniValue::VARR);
|
||||
if (isTAddress || !filterByAddress) {
|
||||
for (int i = 0; i < wtx.vout.size(); i++) {
|
||||
const CTxOut& txout = wtx.vout[i];
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
CTxDestination address;
|
||||
ExtractDestination(txout.scriptPubKey, address);
|
||||
if(IsMine(*pwalletMain, address)){
|
||||
obj.push_back(Pair("address",EncodeDestination(address)));
|
||||
obj.push_back(Pair("scriptPubKey",HexStr(txout.scriptPubKey.begin(), txout.scriptPubKey.end())));
|
||||
obj.push_back(Pair("amount",ValueFromAmount(txout.nValue)));
|
||||
obj.push_back(Pair("vout", i));
|
||||
if (address == tAddress || !filterByAddress) {
|
||||
totalReceived += CAmount(txout.nValue);
|
||||
tReceived.push_back(obj);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
received.push_back(Pair("transparentReceived",tReceived));
|
||||
|
||||
|
||||
//Sapling Sends belonging to the wallet
|
||||
UniValue zsReceived(UniValue::VARR);
|
||||
if (isZsAddress || !filterByAddress) {
|
||||
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
|
||||
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
bool changeTx = false;
|
||||
//Decrypt sapling incoming commitments using IVK
|
||||
std::set<libzcash::SaplingPaymentAddress> addresses;
|
||||
pwalletMain->GetSaplingPaymentAddresses(addresses);
|
||||
for (auto addr : addresses) {
|
||||
libzcash::SaplingExtendedSpendingKey extsk;
|
||||
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
|
||||
auto pt = libzcash::SaplingNotePlaintext::decrypt(
|
||||
outputDesc.encCiphertext, extsk.expsk.full_viewing_key().in_viewing_key(), outputDesc.ephemeralKey, outputDesc.cm);
|
||||
|
||||
if (pt) {
|
||||
auto note = pt.get();
|
||||
auto pt_unwrapped = pt.get();
|
||||
auto memo = pt_unwrapped.memo();
|
||||
obj.push_back(Pair("address",EncodePaymentAddress(addr)));
|
||||
obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value()))));
|
||||
obj.push_back(Pair("memo", HexStr(memo)));
|
||||
|
||||
if (memo[0] <= 0xf4) {
|
||||
auto end = std::find_if(memo.rbegin(), memo.rend(), [](unsigned char v) { return v != 0; });
|
||||
std::string memoStr(memo.begin(), end.base());
|
||||
if (utf8::is_valid(memoStr)) {
|
||||
obj.push_back(Pair("memoStr", memoStr));
|
||||
}
|
||||
}
|
||||
obj.push_back(Pair("shieldedOutputIndex",i));
|
||||
|
||||
//Check Change Status
|
||||
if (wtx.vShieldedSpend.size()!=0) {
|
||||
std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
|
||||
nullifierSet = pwalletMain->GetNullifiersForAddresses({addr});
|
||||
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
|
||||
if (nullifierSet.count(std::make_pair(addr, spendDesc.nullifier))) {
|
||||
changeTx = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
obj.push_back(Pair("change",changeTx));
|
||||
if (addr == zsAddress || !filterByAddress) {
|
||||
totalReceived += CAmount(note.value());
|
||||
zsReceived.push_back(obj);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
received.push_back(Pair("saplingReceived",zsReceived));
|
||||
|
||||
received.push_back(Pair("totalReceived",ValueFromAmount(totalReceived)));
|
||||
}
|
||||
|
||||
|
||||
static void Legacy_zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress) {
|
||||
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
|
||||
//Used to identify incomplete key sets
|
||||
int shieldedOutputCount = 0;
|
||||
int shieldedOutputDecryptedCount = 0;
|
||||
|
||||
//Check address
|
||||
bool isTAddress = false;
|
||||
bool isZsAddress = false;
|
||||
|
||||
CTxDestination tAddress = DecodeDestination(strAddress);
|
||||
auto zAddress = DecodePaymentAddress(strAddress);
|
||||
SaplingPaymentAddress zsAddress;
|
||||
|
||||
if (filterByAddress) {
|
||||
|
||||
if (IsValidDestination(tAddress))
|
||||
isTAddress = true;
|
||||
|
||||
if (IsValidPaymentAddress(zAddress)) {
|
||||
if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
|
||||
zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
|
||||
isZsAddress = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//All Transparent Sends in the transaction
|
||||
|
||||
UniValue tSends(UniValue::VARR);
|
||||
if (isTAddress || !filterByAddress) {
|
||||
for (int i = 0; i < wtx.vout.size(); i++) {
|
||||
const CTxOut& txout = wtx.vout[i];
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
CTxDestination address;
|
||||
ExtractDestination(txout.scriptPubKey, address);
|
||||
obj.push_back(Pair("address",EncodeDestination(address)));
|
||||
obj.push_back(Pair("scriptPubKey",HexStr(txout.scriptPubKey.begin(), txout.scriptPubKey.end())));
|
||||
obj.push_back(Pair("amount",ValueFromAmount(-txout.nValue)));
|
||||
obj.push_back(Pair("vout", i));
|
||||
if (address == tAddress || !filterByAddress) {
|
||||
totalSends += CAmount(-txout.nValue);
|
||||
tSends.push_back(obj);
|
||||
}
|
||||
}
|
||||
}
|
||||
sends.push_back(Pair("transparentSends",tSends));
|
||||
|
||||
//All Shielded Sends in the transaction
|
||||
UniValue zsSends(UniValue::VARR);
|
||||
if (isZsAddress || !filterByAddress) {
|
||||
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
|
||||
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
|
||||
shieldedOutputCount++;
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
bool changeTx = false;
|
||||
|
||||
//Decrypt sapling outgoing t to z transaction using HDseed
|
||||
if (wtx.vShieldedSpend.size()==0) {
|
||||
HDSeed seed;
|
||||
if (pwalletMain->GetHDSeedForDerivation(seed)) {
|
||||
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
|
||||
outputDesc.outCiphertext,ovkForShieldingFromTaddr(seed),outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
|
||||
|
||||
if (opt) {
|
||||
auto opt_unwrapped = opt.get();
|
||||
auto pt = libzcash::SaplingNotePlaintext::decrypt(
|
||||
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
|
||||
|
||||
if (pt) {
|
||||
shieldedOutputDecryptedCount++;
|
||||
auto pt_unwrapped = pt.get();
|
||||
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
|
||||
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
|
||||
obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value()))));
|
||||
obj.push_back(Pair("shieldedOutputIndex",i));
|
||||
obj.push_back(Pair("change",false));
|
||||
if (sentAddr == zsAddress || !filterByAddress) {
|
||||
totalSends += CAmount(-pt_unwrapped.value());
|
||||
zsSends.push_back(obj);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
//attempt Decryption of Outgoing Sapling using wallet extended spending keys
|
||||
} else {
|
||||
std::set<libzcash::SaplingPaymentAddress> addresses;
|
||||
pwalletMain->GetSaplingPaymentAddresses(addresses);
|
||||
for (auto addr : addresses) {
|
||||
libzcash::SaplingExtendedSpendingKey extsk;
|
||||
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
|
||||
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
|
||||
outputDesc.outCiphertext,extsk.expsk.full_viewing_key().ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
|
||||
|
||||
if (opt) {
|
||||
auto opt_unwrapped = opt.get();
|
||||
auto pt = libzcash::SaplingNotePlaintext::decrypt(
|
||||
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
|
||||
|
||||
if (pt) {
|
||||
auto pt_unwrapped = pt.get();
|
||||
auto memo = pt_unwrapped.memo();
|
||||
shieldedOutputDecryptedCount++;
|
||||
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
|
||||
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
|
||||
obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value()))));
|
||||
obj.push_back(Pair("memo", HexStr(memo)));
|
||||
|
||||
if (memo[0] <= 0xf4) {
|
||||
auto end = std::find_if(memo.rbegin(), memo.rend(), [](unsigned char v) { return v != 0; });
|
||||
std::string memoStr(memo.begin(), end.base());
|
||||
if (utf8::is_valid(memoStr)) {
|
||||
obj.push_back(Pair("memoStr", memoStr));
|
||||
}
|
||||
}
|
||||
obj.push_back(Pair("shieldedOutputIndex",i));
|
||||
|
||||
//Check Change Status
|
||||
if (wtx.vShieldedSpend.size()!=0) {
|
||||
std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
|
||||
nullifierSet = pwalletMain->GetNullifiersForAddresses({sentAddr});
|
||||
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
|
||||
if (nullifierSet.count(std::make_pair(sentAddr, spendDesc.nullifier))) {
|
||||
changeTx = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
obj.push_back(Pair("change",changeTx));
|
||||
if (sentAddr == zsAddress || !filterByAddress) {
|
||||
totalSends += CAmount(-pt_unwrapped.value());
|
||||
zsSends.push_back(obj);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
sends.push_back(Pair("saplingSends",zsSends));
|
||||
|
||||
if (shieldedOutputCount != shieldedOutputDecryptedCount) {
|
||||
sends.push_back(Pair("missingSaplingOVK", true));
|
||||
} else {
|
||||
sends.push_back(Pair("missingSaplingOVK", false));
|
||||
}
|
||||
|
||||
sends.push_back(Pair("totalSends",ValueFromAmount(totalSends)));
|
||||
}
|
||||
|
||||
|
||||
static void Legacy_zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType) {
|
||||
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
|
||||
//Track total wallet spend and received
|
||||
CAmount totalSpends = 0;
|
||||
CAmount filteredSpends = 0;
|
||||
CAmount totalReceived = 0;
|
||||
CAmount totalSends = 0;
|
||||
|
||||
//Various Univalue to be added to the final transaction
|
||||
UniValue spends(UniValue::VOBJ);
|
||||
UniValue received(UniValue::VOBJ);
|
||||
UniValue sends(UniValue::VOBJ);
|
||||
UniValue tx(UniValue::VOBJ);
|
||||
|
||||
//Begin Compiling the Decrypted Transaction
|
||||
tx.push_back(Pair("txid", wtx.GetHash().ToString()));
|
||||
|
||||
if(fZdebug)
|
||||
fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
|
||||
|
||||
if (wtx.IsCoinBase())
|
||||
{
|
||||
tx.push_back(Pair("coinbase", true));
|
||||
if (wtx.GetDepthInMainChain() < 1)
|
||||
tx.push_back(Pair("category", "orphan"));
|
||||
else if (wtx.GetBlocksToMaturity() > 0)
|
||||
tx.push_back(Pair("category", "immature"));
|
||||
else
|
||||
tx.push_back(Pair("category", "generate"));
|
||||
} else {
|
||||
tx.push_back(Pair("coinbase", false));
|
||||
tx.push_back(Pair("category", "standard"));
|
||||
}
|
||||
|
||||
tx.push_back(Pair("blockhash", wtx.hashBlock.GetHex()));
|
||||
tx.push_back(Pair("blockindex", wtx.nIndex));
|
||||
int confirms = wtx.GetDepthInMainChain();
|
||||
if(confirms > 0)
|
||||
{
|
||||
tx.push_back(Pair("blocktime", mapBlockIndex[wtx.hashBlock]->GetBlockTime()));
|
||||
} else {
|
||||
tx.push_back(Pair("blocktime", 0));
|
||||
}
|
||||
tx.push_back(Pair("expiryheight", (int64_t)wtx.nExpiryHeight));
|
||||
tx.push_back(Pair("confirmations", confirms));
|
||||
tx.push_back(Pair("time", wtx.GetTxTime()));
|
||||
tx.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
|
||||
|
||||
//Wallet Conflicts
|
||||
UniValue conflicts(UniValue::VARR);
|
||||
BOOST_FOREACH(const uint256& conflict, wtx.GetConflicts())
|
||||
conflicts.push_back(conflict.GetHex());
|
||||
tx.push_back(Pair("walletconflicts", conflicts));
|
||||
|
||||
// Return Type used to determine what is included in the transaction
|
||||
// 0 Spends, Received and spends
|
||||
// 1 Sends
|
||||
// 2 Received
|
||||
// 3 Spends
|
||||
// TODO - Setup Enum, maybe...
|
||||
|
||||
// Add Spends retrieved earlier
|
||||
if (returnType != 2) {
|
||||
Legacy_zsTxSpendsToJSON(wtx, spends, totalSpends, filteredSpends, strAddress, fBool);
|
||||
if ((!fBool || filteredSpends != 0) && (returnType == 0 || returnType == 1)) {
|
||||
tx.push_back(Pair("spends",spends));
|
||||
}
|
||||
}
|
||||
// Get Received
|
||||
if (returnType == 0 || returnType == 2) {
|
||||
Legacy_zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool);
|
||||
if (!fBool || totalReceived != 0) {
|
||||
tx.push_back(Pair("received",received));
|
||||
}
|
||||
}
|
||||
|
||||
// Get Sends
|
||||
if (returnType == 0 || returnType == 3) {
|
||||
//Only include sends if there are spends that belong to the wallet.
|
||||
if (totalSpends != 0 || fBool) {
|
||||
Legacy_zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool);
|
||||
}
|
||||
if (!fBool || totalSends != 0) {
|
||||
tx.push_back(Pair("sends",sends));
|
||||
}
|
||||
}
|
||||
|
||||
if ((returnType == 0 && (!fBool || filteredSpends != 0 || totalReceived != 0 || totalSends != 0))
|
||||
|| (returnType == 1 && (!fBool || filteredSpends != 0))
|
||||
|| (returnType == 2 && (!fBool || totalReceived != 0))
|
||||
|| (returnType == 3 && (!fBool || totalSends != 0))) {
|
||||
ret.push_back(tx);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
std::array<unsigned char, HUSH_MEMO_SIZE> Memo(const std::string& text)
|
||||
{
|
||||
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
|
||||
memo.fill(0);
|
||||
if (text == "empty") // the "no memo" encoding: 0xF6 then zeros
|
||||
memo[0] = 0xF6;
|
||||
else if (text == "binary") // starts below 0xF5 but isn't UTF-8
|
||||
memo[0] = 0xC3, memo[1] = 0x28;
|
||||
else
|
||||
memcpy(memo.data(), text.data(), text.size());
|
||||
return memo;
|
||||
}
|
||||
|
||||
// A real encrypted Sapling output, and when ovk is given, an outCiphertext the sender can decrypt with it
|
||||
// (the wallet never checks proofs or value commitments).
|
||||
OutputDescription Output(const SaplingPaymentAddress& to, uint64_t value, const std::string& memo,
|
||||
const boost::optional<uint256>& ovk = boost::none)
|
||||
{
|
||||
SaplingNote note(to, value);
|
||||
auto enc = SaplingNotePlaintext(note, Memo(memo)).encrypt(note.pk_d);
|
||||
assert(static_cast<bool>(enc));
|
||||
OutputDescription od;
|
||||
od.cv = GetRandHash();
|
||||
od.cm = note.cm().get();
|
||||
od.ephemeralKey = enc->second.get_epk();
|
||||
od.encCiphertext = enc->first;
|
||||
if (ovk) {
|
||||
SaplingOutgoingPlaintext out(note.pk_d, enc->second.get_esk());
|
||||
od.outCiphertext = out.encrypt(*ovk, od.cv, od.cm, enc->second);
|
||||
} else {
|
||||
GetRandBytes(od.outCiphertext.begin(), od.outCiphertext.size());
|
||||
}
|
||||
return od;
|
||||
}
|
||||
|
||||
CTransaction SaplingTx(const std::vector<OutputDescription>& outputs, const std::vector<uint256>& spentNullifiers,
|
||||
const std::vector<COutPoint>& vin = {})
|
||||
{
|
||||
CMutableTransaction mtx;
|
||||
mtx.fOverwintered = true;
|
||||
mtx.nVersion = SAPLING_TX_VERSION;
|
||||
mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
|
||||
mtx.vShieldedOutput = outputs;
|
||||
for (const uint256& nf : spentNullifiers) {
|
||||
SpendDescription sd;
|
||||
sd.nullifier = nf;
|
||||
mtx.vShieldedSpend.push_back(sd);
|
||||
}
|
||||
for (const COutPoint& prev : vin)
|
||||
mtx.vin.push_back(CTxIn(prev));
|
||||
return CTransaction(mtx);
|
||||
}
|
||||
|
||||
SaplingExtendedSpendingKey Key(uint32_t account)
|
||||
{
|
||||
std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x6b);
|
||||
HDSeed seed(raw);
|
||||
return SaplingExtendedSpendingKey::Master(seed).Derive(account | ZIP32_HARDENED_KEY_LIMIT);
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// A wallet whose txs exercise every branch of the history helpers: notes received on default and diversified
|
||||
// addresses, by a viewing-key-only address and by a key imported afterwards; z->z sends with change and memos;
|
||||
// a t->z shielding send; spends of our notes and of someone else's. pwalletMain points at it during the test.
|
||||
class HistoryWallet : public CWallet {
|
||||
public:
|
||||
bool EncryptKeys(CKeyingMaterial& vMasterKeyIn) { return CCryptoKeyStore::EncryptKeys(vMasterKeyIn); }
|
||||
bool Unlock(const CKeyingMaterial& vMasterKeyIn) { return CCryptoKeyStore::Unlock(vMasterKeyIn); }
|
||||
};
|
||||
|
||||
class WalletHistory : public ::testing::Test {
|
||||
protected:
|
||||
HistoryWallet wallet;
|
||||
CWallet* savedWallet = nullptr;
|
||||
SaplingPaymentAddress paA, paA2, paB, paC, paD, paF;
|
||||
CTxDestination taddr;
|
||||
std::map<uint256, int> depthOf;
|
||||
std::vector<uint256> txids;
|
||||
CCoinsView coinsBase;
|
||||
CCoinsViewCache coins{&coinsBase};
|
||||
CCoinsViewCache* savedCoinsTip = nullptr;
|
||||
CBlockIndex* savedTip = nullptr;
|
||||
std::deque<CBlock> blocks;
|
||||
std::deque<CBlockIndex> index;
|
||||
std::vector<uint256> insertedBlocks;
|
||||
|
||||
void SetUp() override {
|
||||
SelectParams(CBaseChainParams::MAIN);
|
||||
savedWallet = pwalletMain;
|
||||
pwalletMain = &wallet;
|
||||
savedCoinsTip = pcoinsTip;
|
||||
pcoinsTip = &coins;
|
||||
LOCK2(cs_main, wallet.cs_wallet);
|
||||
savedTip = chainActive.Tip();
|
||||
|
||||
std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x2d);
|
||||
ASSERT_TRUE(wallet.SetHDSeed(HDSeed(raw)));
|
||||
HDSeed seed;
|
||||
ASSERT_TRUE(wallet.GetHDSeedForDerivation(seed));
|
||||
const uint256 seedOvk = ovkForShieldingFromTaddr(seed);
|
||||
|
||||
auto skA = Key(0), skB = Key(1), skC = Key(2), skD = Key(3), skF = Key(9);
|
||||
paA = skA.DefaultAddress();
|
||||
diversifier_index_t j;
|
||||
j.begin()[0] = 50;
|
||||
paA2 = skA.ToXFVK().Address(j)->second;
|
||||
paB = skB.DefaultAddress();
|
||||
paC = skC.DefaultAddress();
|
||||
paD = skD.DefaultAddress();
|
||||
paF = skF.DefaultAddress();
|
||||
const uint256 ovkA = skA.expsk.full_viewing_key().ovk, ovkC = skC.expsk.full_viewing_key().ovk;
|
||||
ASSERT_TRUE(wallet.AddSaplingZKey(skA, paA));
|
||||
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(skA.expsk.full_viewing_key().in_viewing_key(), paA2));
|
||||
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(skB.expsk.full_viewing_key().in_viewing_key(), paB));
|
||||
ASSERT_TRUE(wallet.AddSaplingZKey(skC, paC));
|
||||
|
||||
CKey tkey;
|
||||
tkey.MakeNewKey(true);
|
||||
ASSERT_TRUE(wallet.AddKeyPubKey(tkey, tkey.GetPubKey()));
|
||||
taddr = tkey.GetPubKey().GetID();
|
||||
|
||||
const uint256 nfA = uint256S("a1"), nfA2 = uint256S("a2"), nfC = uint256S("c1"), nfForeign = uint256S("ff");
|
||||
|
||||
// T0: transparent funds to our taddr.
|
||||
CMutableTransaction t0;
|
||||
t0.vin.push_back(CTxIn(COutPoint(GetRandHash(), 0)));
|
||||
t0.vout.push_back(CTxOut(10 * COIN, GetScriptForDestination(taddr)));
|
||||
AddTx(CTransaction(t0), 30);
|
||||
// T1: received by A, A2, B (viewing key only), C and someone else; no spends.
|
||||
uint256 t1 = AddTx(SaplingTx({Output(paA, 1000, "for A"), Output(paA2, 2000, "empty"), Output(paB, 3000, "for B"),
|
||||
Output(paC, 4000, "binary"), Output(paF, 5000, "not ours")}, {}),
|
||||
25, {{0, nfA}, {1, nfA2}, {3, nfC}});
|
||||
(void)t1;
|
||||
// T2: A sends to F with change back to A, spending A's note: outCiphertexts under A's ovk.
|
||||
AddTx(SaplingTx({Output(paF, 600, "payment", ovkA), Output(paA, 399, "change", ovkA)}, {nfA}), 20);
|
||||
// T3: t->z shielding of T0's output to A, outCiphertext under the seed's shielding ovk.
|
||||
AddTx(SaplingTx({Output(paA, 9 * COIN, "empty", seedOvk)}, {}, {COutPoint(txids[0], 0)}), 15);
|
||||
// T4: C spends its note: to A2 (another of our keys) and change to C.
|
||||
AddTx(SaplingTx({Output(paA2, 1500, "to a diversified A", ovkC), Output(paC, 2499, "binary", ovkC)}, {nfC}), 10);
|
||||
// T5: someone else's spend paying A.
|
||||
AddTx(SaplingTx({Output(paA, 777, "from a stranger")}, {nfForeign}), 8);
|
||||
// T6: received by D before D's key is in the wallet (an import without rescan), then mined.
|
||||
AddTx(SaplingTx({Output(paD, 4242, "for D, found later")}, {}), 5);
|
||||
// T8: spends one of our notes and one of someone else's (a coinjoin-like tx): missingSpendingKeys.
|
||||
AddTx(SaplingTx({Output(paF, 100, "mixed", ovkA)}, {nfA, uint256S("fe")}), 3);
|
||||
// T9: spends A2's note, but its one output's outCiphertext is under nobody's ovk we know: missingSaplingOVK.
|
||||
AddTx(SaplingTx({Output(paF, 50, "unreadable send")}, {nfA2}), 2);
|
||||
// T7: spends A2's note (A and A2 share A's key); never mined and not in the mempool (depth -1).
|
||||
AddTx(SaplingTx({Output(paF, 2900, "", ovkA)}, {nfA2}), 0);
|
||||
ASSERT_TRUE(wallet.AddSaplingZKey(skD, paD));
|
||||
|
||||
Mine();
|
||||
}
|
||||
|
||||
void TearDown() override {
|
||||
LOCK(cs_main);
|
||||
chainActive.SetTip(savedTip);
|
||||
for (const uint256& h : insertedBlocks)
|
||||
mapBlockIndex.erase(h);
|
||||
pcoinsTip = savedCoinsTip;
|
||||
pwalletMain = savedWallet;
|
||||
}
|
||||
|
||||
uint256 AddTx(const CTransaction& tx, int depth, const std::map<uint32_t, uint256>& nullifiers = {}) {
|
||||
CWalletTx wtx(&wallet, tx);
|
||||
mapSaplingNoteData_t found = wallet.FindMySaplingNotes(tx).first;
|
||||
for (const auto& nf : nullifiers)
|
||||
found.at(SaplingOutPoint(tx.GetHash(), nf.first)).nullifier = nf.second;
|
||||
wtx.SetSaplingNoteData(found);
|
||||
EXPECT_TRUE(wallet.AddToWallet(wtx, true, nullptr));
|
||||
if (depth > 0)
|
||||
depthOf[tx.GetHash()] = depth;
|
||||
txids.push_back(tx.GetHash());
|
||||
return tx.GetHash();
|
||||
}
|
||||
|
||||
void Mine() {
|
||||
const int tip = 40;
|
||||
for (int h = 0; h <= tip; h++) {
|
||||
blocks.emplace_back();
|
||||
CBlock& block = blocks.back();
|
||||
block.nTime = 1700000000 + 36 * h;
|
||||
block.nNonce = uint256S(strprintf("%x", 0x900 + h));
|
||||
if (h > 0)
|
||||
block.hashPrevBlock = blocks[h - 1].GetHash();
|
||||
for (const auto& d : depthOf)
|
||||
if (tip - d.second + 1 == h)
|
||||
block.vtx.push_back(wallet.mapWallet.at(d.first));
|
||||
block.hashMerkleRoot = block.BuildMerkleTree();
|
||||
index.emplace_back(block);
|
||||
CBlockIndex& bi = index.back();
|
||||
bi.SetHeight(h);
|
||||
bi.pprev = h > 0 ? &index[h - 1] : nullptr;
|
||||
auto inserted = mapBlockIndex.insert(std::make_pair(block.GetHash(), &bi));
|
||||
ASSERT_TRUE(inserted.second);
|
||||
insertedBlocks.push_back(block.GetHash());
|
||||
bi.phashBlock = &inserted.first->first;
|
||||
}
|
||||
chainActive.SetTip(&index[tip]);
|
||||
for (const auto& d : depthOf)
|
||||
wallet.mapWallet.at(d.first).SetMerkleBranch(blocks[tip - d.second + 1]);
|
||||
}
|
||||
|
||||
std::string NewJSON(const CWalletTx& wtx, const std::string& addr, bool fBool, int returnType) {
|
||||
UniValue ret(UniValue::VARR);
|
||||
zsWalletTxJSON(wtx, ret, addr, fBool, returnType, GetSaplingHistoryKeys());
|
||||
return ret.write();
|
||||
}
|
||||
std::string LegacyJSON(const CWalletTx& wtx, const std::string& addr, bool fBool, int returnType) {
|
||||
UniValue ret(UniValue::VARR);
|
||||
Legacy_zsWalletTxJSON(wtx, ret, addr, fBool, returnType);
|
||||
return ret.write();
|
||||
}
|
||||
void ExpectMatchesLegacy(bool checkCoverage = true);
|
||||
};
|
||||
|
||||
// Every tx, every return type, unfiltered and filtered by each kind of address: byte-identical JSON. Run with
|
||||
// the wallet's plaintext cache warm (as after the startup warm-up) and cold (every output trial-decrypted).
|
||||
class WalletHistoryCold : public WalletHistory {};
|
||||
|
||||
TEST_F(WalletHistoryCold, MatchesLegacy) {
|
||||
LOCK2(cs_main, wallet.cs_wallet);
|
||||
for (auto& p : wallet.mapWallet)
|
||||
for (auto& item : p.second.mapSaplingNoteData)
|
||||
item.second.cachedPlaintext = boost::none;
|
||||
ExpectMatchesLegacy();
|
||||
}
|
||||
|
||||
TEST_F(WalletHistory, MatchesLegacy) {
|
||||
LOCK2(cs_main, wallet.cs_wallet);
|
||||
ExpectMatchesLegacy();
|
||||
}
|
||||
|
||||
void WalletHistory::ExpectMatchesLegacy(bool checkCoverage) {
|
||||
const std::vector<std::pair<std::string, bool>> filters = {
|
||||
{"*", false}, {"*", true}, {EncodePaymentAddress(paA), true}, {EncodePaymentAddress(paA2), true},
|
||||
{EncodePaymentAddress(paB), true}, {EncodePaymentAddress(paC), true}, {EncodePaymentAddress(paD), true},
|
||||
{EncodePaymentAddress(paF), true}, {EncodeDestination(taddr), true},
|
||||
};
|
||||
std::string all;
|
||||
for (const uint256& txid : txids) {
|
||||
const CWalletTx& wtx = wallet.mapWallet.at(txid);
|
||||
for (int returnType = 0; returnType <= 3; returnType++) {
|
||||
for (const auto& f : filters) {
|
||||
// New first: the legacy spends helper default-inserts into wallet maps.
|
||||
std::string got = NewJSON(wtx, f.first, f.second, returnType);
|
||||
std::string want = LegacyJSON(wtx, f.first, f.second, returnType);
|
||||
EXPECT_EQ(want, got) << "tx " << txid.ToString() << " returnType " << returnType << " filter " << f.first << "/" << f.second;
|
||||
all += got;
|
||||
}
|
||||
}
|
||||
}
|
||||
// The fixture really reaches the interesting branches.
|
||||
if (!checkCoverage)
|
||||
return;
|
||||
for (const char* needle : {"\"change\":true", "\"memoStr\"", "\"saplingSpends\":[{", "\"transparentSpends\":[{",
|
||||
"\"saplingSends\":[{", "\"saplingReceived\":[{", "\"missingSaplingOVK\":true",
|
||||
"\"missingSpendingKeys\":true", "for D, found later"}) {
|
||||
EXPECT_NE(std::string::npos, all.find(needle)) << needle;
|
||||
}
|
||||
}
|
||||
|
||||
// The same comparison on an encrypted wallet, locked (no spending keys or HD seed available: no shielded
|
||||
// received or sent entries, as before) and then unlocked.
|
||||
class WalletHistoryEncrypted : public WalletHistory {};
|
||||
|
||||
TEST_F(WalletHistoryEncrypted, MatchesLegacyLockedAndUnlocked) {
|
||||
LOCK2(cs_main, wallet.cs_wallet);
|
||||
CKeyingMaterial masterKey(32, 0x5e);
|
||||
ASSERT_TRUE(wallet.EncryptKeys(masterKey));
|
||||
ASSERT_TRUE(wallet.IsLocked());
|
||||
ASSERT_TRUE(GetSaplingHistoryKeys().spendable.empty());
|
||||
ExpectMatchesLegacy(false);
|
||||
ASSERT_TRUE(wallet.Unlock(masterKey));
|
||||
ASSERT_FALSE(GetSaplingHistoryKeys().spendable.empty());
|
||||
ExpectMatchesLegacy();
|
||||
}
|
||||
|
||||
// The history helpers only read the wallet. The old spends helper's operator[] lookups inserted an entry into
|
||||
// mapSaplingNullifiersToNotes for every other wallet's nullifier they met, which made a later
|
||||
// z_viewtransaction of that tx fail (map::at on a null outpoint).
|
||||
TEST_F(WalletHistory, DoesNotModifyTheWallet) {
|
||||
LOCK2(cs_main, wallet.cs_wallet);
|
||||
const size_t nullifierEntries = wallet.mapSaplingNullifiersToNotes.size();
|
||||
std::map<uint256, size_t> noteEntries;
|
||||
for (const auto& p : wallet.mapWallet)
|
||||
noteEntries[p.first] = p.second.mapSaplingNoteData.size();
|
||||
|
||||
for (const uint256& txid : txids)
|
||||
for (int returnType = 0; returnType <= 3; returnType++)
|
||||
NewJSON(wallet.mapWallet.at(txid), "*", false, returnType);
|
||||
EXPECT_EQ(nullifierEntries, wallet.mapSaplingNullifiersToNotes.size());
|
||||
EXPECT_EQ(0u, wallet.mapSaplingNullifiersToNotes.count(uint256S("ff"))) << "someone else's spend";
|
||||
for (const auto& p : wallet.mapWallet)
|
||||
EXPECT_EQ(noteEntries.at(p.first), p.second.mapSaplingNoteData.size());
|
||||
|
||||
// What the old helpers did, for the record.
|
||||
for (const uint256& txid : txids)
|
||||
LegacyJSON(wallet.mapWallet.at(txid), "*", false, 0);
|
||||
EXPECT_EQ(1u, wallet.mapSaplingNullifiersToNotes.count(uint256S("ff")));
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,272 +0,0 @@
|
||||
// Copyright (c) 2016-2024 The Hush developers
|
||||
// Distributed under the GPLv3 software license, see the accompanying
|
||||
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <deque>
|
||||
|
||||
#include "chain.h"
|
||||
#include "coins.h"
|
||||
#include "main.h"
|
||||
#include "primitives/block.h"
|
||||
#include "primitives/transaction.h"
|
||||
#include "random.h"
|
||||
#include "uint256.h"
|
||||
#include "wallet/wallet.h"
|
||||
#include "zcash/Note.hpp"
|
||||
#include "zcash/zip32.h"
|
||||
|
||||
namespace {
|
||||
|
||||
class NoteDataWallet : public CWallet {
|
||||
public:
|
||||
bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock = true) {
|
||||
return CWallet::UpdatedNoteData(wtxIn, wtx, fSameBlock);
|
||||
}
|
||||
int VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessOnly) {
|
||||
return CWallet::VerifyAndSetInitialWitness(pindex, witnessOnly);
|
||||
}
|
||||
};
|
||||
|
||||
libzcash::SaplingIncomingViewingKey TestIvk()
|
||||
{
|
||||
return libzcash::SaplingIncomingViewingKey(uint256S("0101010101010101010101010101010101010101010101010101010101010101"));
|
||||
}
|
||||
|
||||
// A note as the wallet holds it once witnessed: nullifier set, witnesses cached at a height.
|
||||
SaplingNoteData WitnessedNote(const SaplingMerkleTree& tree, int height, const uint256& nullifier)
|
||||
{
|
||||
SaplingNoteData nd(TestIvk(), nullifier);
|
||||
nd.witnesses.push_front(tree.witness());
|
||||
nd.witnessHeight = height;
|
||||
return nd;
|
||||
}
|
||||
|
||||
// A note as FindMySaplingNotes returns it during a replay or rescan: no nullifier, no witnesses.
|
||||
SaplingNoteData FreshNote()
|
||||
{
|
||||
return SaplingNoteData(TestIvk());
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// A replayed or rescanned tx must keep the witnesses AND the nullifier it already had, and must not
|
||||
// report an update: dropping the nullifier made the next block clear and reseed every carried witness.
|
||||
TEST(WalletNoteData, ReplayKeepsNullifierAndWitnesses) {
|
||||
NoteDataWallet wallet;
|
||||
CWalletTx wtx(&wallet, CTransaction());
|
||||
uint256 txid = uint256S("aa");
|
||||
SaplingOutPoint op0(txid, 0);
|
||||
|
||||
SaplingMerkleTree tree;
|
||||
tree.append(uint256S("01"));
|
||||
const uint256 nf = uint256S("f00d");
|
||||
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 100, nf);
|
||||
wtx.mapSaplingNoteData[op0].witnessRootValidated = true;
|
||||
auto before = wtx.mapSaplingNoteData;
|
||||
|
||||
CWalletTx replay(&wallet, CTransaction());
|
||||
replay.mapSaplingNoteData[op0] = FreshNote();
|
||||
|
||||
EXPECT_FALSE(wallet.UpdatedNoteData(replay, wtx));
|
||||
ASSERT_EQ(1u, wtx.mapSaplingNoteData.size());
|
||||
const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
|
||||
ASSERT_TRUE(static_cast<bool>(nd.nullifier));
|
||||
EXPECT_EQ(nf, *nd.nullifier);
|
||||
EXPECT_EQ(100, nd.witnessHeight);
|
||||
EXPECT_TRUE(nd.witnesses == before.at(op0).witnesses);
|
||||
// ...but the kept witness is re-checked against the active chain once (in-memory flag only).
|
||||
EXPECT_FALSE(nd.witnessRootValidated);
|
||||
}
|
||||
|
||||
// A tx re-mined in a different block (reorg) has its notes at new tree positions: the old nullifier
|
||||
// must not be carried, so VerifyAndSetInitialWitness reseeds the note at its new position.
|
||||
TEST(WalletNoteData, MovedTxDropsNullifier) {
|
||||
NoteDataWallet wallet;
|
||||
CWalletTx wtx(&wallet, CTransaction());
|
||||
SaplingOutPoint op0(uint256S("ee"), 0);
|
||||
|
||||
SaplingMerkleTree tree;
|
||||
tree.append(uint256S("04"));
|
||||
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 500, uint256S("cafe"));
|
||||
|
||||
CWalletTx moved(&wallet, CTransaction());
|
||||
moved.mapSaplingNoteData[op0] = FreshNote();
|
||||
|
||||
EXPECT_TRUE(wallet.UpdatedNoteData(moved, wtx, false));
|
||||
const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
|
||||
EXPECT_FALSE(static_cast<bool>(nd.nullifier));
|
||||
}
|
||||
|
||||
// When the incoming data really adds something (a note for a newly added key), the tx is updated,
|
||||
// and the notes it already had keep their witnesses, witness height and nullifier together.
|
||||
TEST(WalletNoteData, NewNoteUpdatesAndKeepsExistingNoteState) {
|
||||
NoteDataWallet wallet;
|
||||
CWalletTx wtx(&wallet, CTransaction());
|
||||
uint256 txid = uint256S("bb");
|
||||
SaplingOutPoint op0(txid, 0);
|
||||
SaplingOutPoint op1(txid, 1);
|
||||
|
||||
SaplingMerkleTree tree;
|
||||
tree.append(uint256S("02"));
|
||||
const uint256 nf = uint256S("beef");
|
||||
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 200, nf);
|
||||
|
||||
CWalletTx incoming(&wallet, CTransaction());
|
||||
incoming.mapSaplingNoteData[op0] = FreshNote();
|
||||
incoming.mapSaplingNoteData[op1] = FreshNote();
|
||||
|
||||
EXPECT_TRUE(wallet.UpdatedNoteData(incoming, wtx));
|
||||
ASSERT_EQ(2u, wtx.mapSaplingNoteData.size());
|
||||
const SaplingNoteData& nd0 = wtx.mapSaplingNoteData.at(op0);
|
||||
ASSERT_TRUE(static_cast<bool>(nd0.nullifier));
|
||||
EXPECT_EQ(nf, *nd0.nullifier);
|
||||
EXPECT_EQ(200, nd0.witnessHeight);
|
||||
EXPECT_EQ(1u, nd0.witnesses.size());
|
||||
const SaplingNoteData& nd1 = wtx.mapSaplingNoteData.at(op1);
|
||||
EXPECT_FALSE(static_cast<bool>(nd1.nullifier));
|
||||
EXPECT_TRUE(nd1.witnesses.empty());
|
||||
EXPECT_EQ(-1, nd1.witnessHeight);
|
||||
}
|
||||
|
||||
// Nothing is carried for a note that has no witnesses (PART B2): its height and nullifier describe
|
||||
// no witness, so they must not survive onto the incoming data.
|
||||
TEST(WalletNoteData, UnwitnessedNoteCarriesNothing) {
|
||||
NoteDataWallet wallet;
|
||||
CWalletTx wtx(&wallet, CTransaction());
|
||||
SaplingOutPoint op0(uint256S("cc"), 0);
|
||||
|
||||
SaplingNoteData stale(TestIvk(), uint256S("dead"));
|
||||
stale.witnessHeight = 300;
|
||||
wtx.mapSaplingNoteData[op0] = stale;
|
||||
|
||||
CWalletTx incoming(&wallet, CTransaction());
|
||||
incoming.mapSaplingNoteData[op0] = FreshNote();
|
||||
|
||||
EXPECT_TRUE(wallet.UpdatedNoteData(incoming, wtx));
|
||||
const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
|
||||
EXPECT_FALSE(static_cast<bool>(nd.nullifier));
|
||||
EXPECT_EQ(-1, nd.witnessHeight);
|
||||
EXPECT_TRUE(nd.witnesses.empty());
|
||||
}
|
||||
|
||||
// Incoming data with no Sapling notes never touches the wallet's note data.
|
||||
TEST(WalletNoteData, EmptyIncomingIsNoUpdate) {
|
||||
NoteDataWallet wallet;
|
||||
CWalletTx wtx(&wallet, CTransaction());
|
||||
SaplingOutPoint op0(uint256S("dd"), 0);
|
||||
SaplingMerkleTree tree;
|
||||
tree.append(uint256S("03"));
|
||||
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 400, uint256S("abcd"));
|
||||
|
||||
CWalletTx incoming(&wallet, CTransaction());
|
||||
EXPECT_FALSE(wallet.UpdatedNoteData(incoming, wtx));
|
||||
EXPECT_EQ(1u, wtx.mapSaplingNoteData.size());
|
||||
EXPECT_EQ(400, wtx.mapSaplingNoteData.at(op0).witnessHeight);
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
// A real encrypted Sapling output to pa (the wallet never checks its proof or value commitment).
|
||||
OutputDescription EncryptedOutput(const libzcash::SaplingPaymentAddress& pa, uint64_t value)
|
||||
{
|
||||
libzcash::SaplingNote note(pa, value);
|
||||
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
|
||||
memo.fill(0);
|
||||
auto enc = libzcash::SaplingNotePlaintext(note, memo).encrypt(note.pk_d);
|
||||
assert(static_cast<bool>(enc));
|
||||
OutputDescription od;
|
||||
od.cv = GetRandHash();
|
||||
od.cm = note.cm().get();
|
||||
od.ephemeralKey = enc->second.get_epk();
|
||||
od.encCiphertext = enc->first;
|
||||
return od;
|
||||
}
|
||||
|
||||
// A three-block active chain (heights 0-2) whose block 1 holds the given txs, undone on destruction:
|
||||
// chainActive, mapBlockIndex and pcoinsTip are process-wide and every gtest suite shares the process.
|
||||
struct FakeActiveChain {
|
||||
std::deque<CBlock> blocks;
|
||||
std::deque<CBlockIndex> index;
|
||||
CBlockIndex* savedTip;
|
||||
CCoinsView coinsBase;
|
||||
CCoinsViewCache coins{&coinsBase};
|
||||
CCoinsViewCache* savedCoinsTip;
|
||||
|
||||
explicit FakeActiveChain(const std::vector<CTransaction>& inBlock1) {
|
||||
LOCK(cs_main);
|
||||
savedTip = chainActive.Tip();
|
||||
savedCoinsTip = pcoinsTip;
|
||||
pcoinsTip = &coins;
|
||||
for (int h = 0; h <= 2; h++) {
|
||||
blocks.emplace_back();
|
||||
CBlock& block = blocks.back();
|
||||
block.nTime = 1700000000 + h;
|
||||
block.nNonce = uint256S(strprintf("%x", 0x700 + h));
|
||||
if (h > 0)
|
||||
block.hashPrevBlock = blocks[h - 1].GetHash();
|
||||
if (h == 1)
|
||||
block.vtx = inBlock1;
|
||||
block.hashMerkleRoot = block.BuildMerkleTree();
|
||||
index.emplace_back(block);
|
||||
CBlockIndex& bi = index.back();
|
||||
bi.SetHeight(h);
|
||||
bi.pprev = h > 0 ? &index[h - 1] : nullptr;
|
||||
bi.phashBlock = &mapBlockIndex.insert(std::make_pair(block.GetHash(), &bi)).first->first;
|
||||
}
|
||||
chainActive.SetTip(&index.back());
|
||||
}
|
||||
~FakeActiveChain() {
|
||||
LOCK(cs_main);
|
||||
chainActive.SetTip(savedTip);
|
||||
for (const CBlock& block : blocks)
|
||||
mapBlockIndex.erase(block.GetHash());
|
||||
pcoinsTip = savedCoinsTip;
|
||||
}
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
// A note found with an incoming viewing key only (z_importviewingkey) can't be spent and never gets
|
||||
// a nullifier, because deriving one takes the full viewing key. VerifyAndSetInitialWitness must leave
|
||||
// it alone. It used to clear it and reseed its witness from the block on disk on every call, then
|
||||
// abort on the missing nullifier the first time the note was confirmed. Here there are no block
|
||||
// files, so the old reseed path throws.
|
||||
TEST(WalletNoteData, ViewingKeyOnlyNoteIsNotWitnessed) {
|
||||
NoteDataWallet wallet;
|
||||
std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x3c);
|
||||
HDSeed seed(raw);
|
||||
auto sk = libzcash::SaplingExtendedSpendingKey::Master(seed);
|
||||
auto pa = sk.DefaultAddress();
|
||||
|
||||
LOCK2(cs_main, wallet.cs_wallet);
|
||||
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(sk.expsk.full_viewing_key().in_viewing_key(), pa));
|
||||
ASSERT_FALSE(wallet.HaveSaplingFullViewingKey(sk.expsk.full_viewing_key().in_viewing_key()));
|
||||
|
||||
CMutableTransaction mtx;
|
||||
mtx.fOverwintered = true;
|
||||
mtx.nVersion = SAPLING_TX_VERSION;
|
||||
mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
|
||||
mtx.vShieldedOutput.push_back(EncryptedOutput(pa, 5000));
|
||||
CTransaction tx(mtx);
|
||||
|
||||
CWalletTx wtx(&wallet, tx);
|
||||
mapSaplingNoteData_t found = wallet.FindMySaplingNotes(tx).first;
|
||||
ASSERT_EQ(1u, found.size()) << "the viewing key finds the note";
|
||||
wtx.SetSaplingNoteData(found);
|
||||
ASSERT_TRUE(wallet.AddToWallet(wtx, true, nullptr));
|
||||
|
||||
FakeActiveChain chain({tx});
|
||||
CWalletTx& stored = wallet.mapWallet.at(tx.GetHash());
|
||||
stored.SetMerkleBranch(chain.blocks[1]);
|
||||
ASSERT_EQ(2, stored.GetDepthInMainChain());
|
||||
|
||||
int minHeight = -1;
|
||||
ASSERT_NO_THROW(minHeight = wallet.VerifyAndSetInitialWitness(chainActive.Tip(), false));
|
||||
EXPECT_EQ(chainActive.Height(), minHeight) << "it must not pull the witness rebuild down to its height";
|
||||
const SaplingNoteData& nd = stored.mapSaplingNoteData.at(SaplingOutPoint(tx.GetHash(), 0));
|
||||
EXPECT_TRUE(nd.witnesses.empty());
|
||||
EXPECT_FALSE(static_cast<bool>(nd.nullifier));
|
||||
EXPECT_EQ(-1, nd.witnessHeight);
|
||||
}
|
||||
@@ -0,0 +1,848 @@
|
||||
// Copyright (c) 2016-2026 The Hush developers
|
||||
// Distributed under the GPLv3 software license, see the accompanying
|
||||
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
|
||||
//
|
||||
// Proves that the optimised CWallet::GetFilteredNotes returns exactly what the original
|
||||
// did: the same entries, in the same order, with every field equal. The optimisations
|
||||
// are references instead of copies, skipping txs without note data, the spent check
|
||||
// before trial decryption, the exact ivk pre-filter for address-filtered calls, and
|
||||
// building the note and address from decrypt_with_pk_d. RefGetFilteredNotes below is a
|
||||
// verbatim copy of the original body (c1040028e), and both run over the same hand-built
|
||||
// wallet for a grid of filters and parameters. The file also tests
|
||||
// SaplingNotePlaintext::decrypt_with_pk_d against decrypt().
|
||||
//
|
||||
// No proofs are needed: the wallet never verifies them, so notes are encrypted by hand
|
||||
// into fabricated OutputDescriptions and added with AddToWallet(wtx, true, NULL). Depth
|
||||
// is faked with a mapBlockIndex chain (depth >= 1), the mempool (depth 0) or neither
|
||||
// (depth -1). minDepth > 1 is not covered: that branch calls tx_height(), which reads
|
||||
// the tx from disk (the regtest differential covers it).
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include "arith_uint256.h"
|
||||
#include "chain.h"
|
||||
#include "chainparams.h"
|
||||
#include "key_io.h"
|
||||
#include "main.h"
|
||||
#include "primitives/transaction.h"
|
||||
#include "txmempool.h"
|
||||
#include "uint256.h"
|
||||
#include "util.h"
|
||||
#include "wallet/crypter.h"
|
||||
#include "wallet/wallet.h"
|
||||
#include "zcash/Address.hpp"
|
||||
#include "zcash/Note.hpp"
|
||||
#include "zcash/zip32.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <array>
|
||||
#include <cassert>
|
||||
#include <climits>
|
||||
#include <list>
|
||||
#include <memory>
|
||||
#include <set>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include <boost/filesystem.hpp>
|
||||
|
||||
// The original body calls these; wallet.cpp declares them the same way (wallet.cpp:77-78).
|
||||
// Both are defined in main.cpp (via hush.h). Neither is reached here (minDepth <= 1).
|
||||
int32_t hush_dpowconfs(int32_t height,int32_t numconfs);
|
||||
int tx_height( const uint256 &hash );
|
||||
|
||||
/**
|
||||
* Verbatim copy of the original
|
||||
* CWallet::GetFilteredNotes(std::vector<SaplingNoteEntry>&, std::set<PaymentAddress>&,
|
||||
* int, int, bool, bool, bool) at c1040028e (wallet.cpp:5637-5721). The only changes:
|
||||
* it is a free function, so wallet members are `wallet.`-qualified (all of them are
|
||||
* public: cs_wallet, mapWallet, IsSaplingSpent, GetSaplingIncomingViewingKey,
|
||||
* GetSaplingFullViewingKey, HaveSaplingSpendingKey, IsLockedNote), and the two LogPrintf
|
||||
* calls are dropped (they do not affect the output).
|
||||
*/
|
||||
static void RefGetFilteredNotes(
|
||||
CWallet& wallet,
|
||||
std::vector<SaplingNoteEntry>& saplingEntries,
|
||||
std::set<libzcash::PaymentAddress>& filterAddresses,
|
||||
int minDepth,
|
||||
int maxDepth,
|
||||
bool ignoreSpent,
|
||||
bool requireSpendingKey,
|
||||
bool ignoreLocked)
|
||||
{
|
||||
using namespace libzcash;
|
||||
|
||||
LOCK2(cs_main, wallet.cs_wallet);
|
||||
|
||||
for (auto & p : wallet.mapWallet) {
|
||||
CWalletTx wtx = p.second;
|
||||
|
||||
// Filter the transactions before checking for notes
|
||||
if (!CheckFinalTx(wtx) || wtx.GetBlocksToMaturity() > 0)
|
||||
continue;
|
||||
|
||||
if (minDepth > 1) {
|
||||
int nHeight = tx_height(wtx.GetHash());
|
||||
if ( nHeight == 0 ) {
|
||||
continue;
|
||||
}
|
||||
int nDepth = wtx.GetDepthInMainChain();
|
||||
int dpowconfs = hush_dpowconfs(nHeight,nDepth);
|
||||
if ( dpowconfs < minDepth || dpowconfs > maxDepth) {
|
||||
continue;
|
||||
}
|
||||
} else {
|
||||
if ( wtx.GetDepthInMainChain() < minDepth ||
|
||||
wtx.GetDepthInMainChain() > maxDepth) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
for (auto & pair : wtx.mapSaplingNoteData) {
|
||||
SaplingOutPoint op = pair.first;
|
||||
SaplingNoteData nd = pair.second;
|
||||
|
||||
auto maybe_pt = SaplingNotePlaintext::decrypt(
|
||||
wtx.vShieldedOutput[op.n].encCiphertext,
|
||||
nd.ivk,
|
||||
wtx.vShieldedOutput[op.n].ephemeralKey,
|
||||
wtx.vShieldedOutput[op.n].cm);
|
||||
assert(static_cast<bool>(maybe_pt));
|
||||
auto notePt = maybe_pt.get();
|
||||
|
||||
auto maybe_pa = nd.ivk.address(notePt.d);
|
||||
assert(static_cast<bool>(maybe_pa));
|
||||
auto pa = maybe_pa.get();
|
||||
|
||||
// skip notes which belong to a different payment address in the wallet
|
||||
if (!(filterAddresses.empty() || filterAddresses.count(pa))) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (ignoreSpent && nd.nullifier && wallet.IsSaplingSpent(*nd.nullifier)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// skip notes which cannot be spent
|
||||
if (requireSpendingKey) {
|
||||
libzcash::SaplingIncomingViewingKey ivk;
|
||||
libzcash::SaplingFullViewingKey fvk;
|
||||
if (!(wallet.GetSaplingIncomingViewingKey(pa, ivk) &&
|
||||
wallet.GetSaplingFullViewingKey(ivk, fvk) &&
|
||||
wallet.HaveSaplingSpendingKey(fvk))) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
// skip locked notes
|
||||
if (ignoreLocked && wallet.IsLockedNote(op)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
auto note = notePt.note(nd.ivk).get();
|
||||
saplingEntries.push_back(SaplingNoteEntry {
|
||||
op, pa, note, notePt.memo(), wtx.GetDepthInMainChain() });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
// Heights 0..CHAIN_LENGTH-1, so a tx in the block at height h has depth CHAIN_LENGTH - h.
|
||||
const int CHAIN_LENGTH = 10;
|
||||
|
||||
// The Jubjub subgroup order r_J. An ivk + r_J generates the same addresses as ivk,
|
||||
// because librustzcash_ivk_to_pkd multiplies by the raw 256-bit value.
|
||||
const char JUBJUB_R_HEX[] = "0e7db4ea6533afa906673b0101343b00a6682093ccc81082d0970e5ed6f72cb7";
|
||||
|
||||
// Keys and addresses of the test wallet (see WalletScanNotes::BuildWallet).
|
||||
struct TestKeys {
|
||||
libzcash::SaplingFullViewingKey fvk1, fvk2;
|
||||
libzcash::SaplingIncomingViewingKey ivk1, ivk2, ivk3, ivk4, ivkN, aliasIvk1;
|
||||
libzcash::SaplingPaymentAddress a1; // K1 default address (spending key)
|
||||
libzcash::SaplingPaymentAddress d1; // K1 diversified address, recorded correctly
|
||||
libzcash::SaplingPaymentAddress u1; // K1 diversified address, never recorded
|
||||
libzcash::SaplingPaymentAddress x1; // K1 diversified address, recorded with K2's ivk
|
||||
libzcash::SaplingPaymentAddress y1; // K1 diversified address, recorded with ivk1 + r_J
|
||||
libzcash::SaplingPaymentAddress a2; // K2 default address (spending key)
|
||||
libzcash::SaplingPaymentAddress a3; // K3: incoming viewing key only
|
||||
libzcash::SaplingPaymentAddress a4; // K4: full viewing key only
|
||||
libzcash::SaplingPaymentAddress aN; // address of a non-canonical ivk (>= 2^251)
|
||||
libzcash::SaplingPaymentAddress foreign; // not in the wallet at all
|
||||
std::vector<libzcash::SaplingPaymentAddress> empty; // spending keys that hold no notes
|
||||
size_t nNotes;
|
||||
};
|
||||
|
||||
// One output of a fabricated tx that the wallet owns: the address it pays, the ivk the
|
||||
// wallet records for it (SaplingNoteData::ivk) and its cached nullifier, if any.
|
||||
struct OwnedOutput {
|
||||
libzcash::SaplingPaymentAddress addr;
|
||||
libzcash::SaplingIncomingViewingKey ivk;
|
||||
uint64_t value;
|
||||
boost::optional<uint256> nullifier;
|
||||
};
|
||||
|
||||
struct NamedFilter {
|
||||
std::string name;
|
||||
std::set<libzcash::PaymentAddress> addrs;
|
||||
};
|
||||
|
||||
// CCryptoKeyStore::EncryptKeys is protected, and CWallet::EncryptWallet needs a wallet file.
|
||||
class EncryptableWallet : public CWallet {
|
||||
public:
|
||||
bool EncryptKeysForTest(CKeyingMaterial& vMasterKeyIn) { return EncryptKeys(vMasterKeyIn); }
|
||||
};
|
||||
|
||||
libzcash::SaplingExtendedSpendingKey KeyFromSeedByte(unsigned char b)
|
||||
{
|
||||
RawHDSeed raw(32, b);
|
||||
HDSeed seed(raw);
|
||||
return libzcash::SaplingExtendedSpendingKey::Master(seed);
|
||||
}
|
||||
|
||||
// A diversified address of xsk other than its default one: the first valid diversifier
|
||||
// index at or after `start` (the default address uses the first one from 0).
|
||||
libzcash::SaplingPaymentAddress DiversifiedAddress(
|
||||
const libzcash::SaplingExtendedSpendingKey& xsk, unsigned char start)
|
||||
{
|
||||
libzcash::diversifier_index_t j;
|
||||
j.begin()[0] = start;
|
||||
return xsk.ToXFVK().Address(j).get().second;
|
||||
}
|
||||
|
||||
// A synthetic nullifier. GetFilteredNotes only compares nullifiers, it never recomputes them.
|
||||
uint256 Nf(unsigned char n)
|
||||
{
|
||||
uint256 nf;
|
||||
nf.begin()[0] = n;
|
||||
nf.begin()[31] = 0x4e;
|
||||
return nf;
|
||||
}
|
||||
|
||||
// A shielded output paying `value` to `addr`, encrypted the way a real sender does
|
||||
// (SaplingNotePlaintext::encrypt), with a random rcm and a memo derived from the value.
|
||||
// There is no proof, value commitment or outgoing ciphertext: the wallet reads only cm,
|
||||
// ephemeralKey and encCiphertext.
|
||||
OutputDescription MakeOutput(const libzcash::SaplingPaymentAddress& addr, uint64_t value)
|
||||
{
|
||||
libzcash::SaplingNote note(addr, value);
|
||||
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
|
||||
memo.fill(0);
|
||||
memo[0] = value & 0xff;
|
||||
memo[1] = (value >> 8) & 0xff;
|
||||
libzcash::SaplingNotePlaintext pt(note, memo);
|
||||
libzcash::SaplingNotePlaintextEncryptionResult res = pt.encrypt(note.pk_d).get();
|
||||
|
||||
OutputDescription od;
|
||||
od.cv = uint256();
|
||||
od.cm = note.cm().get();
|
||||
od.ephemeralKey = res.second.get_epk();
|
||||
od.encCiphertext = res.first;
|
||||
od.outCiphertext.fill(0);
|
||||
od.zkproof.fill(0);
|
||||
return od;
|
||||
}
|
||||
|
||||
std::set<libzcash::PaymentAddress> Filter(const std::vector<libzcash::SaplingPaymentAddress>& addrs)
|
||||
{
|
||||
std::set<libzcash::PaymentAddress> s;
|
||||
for (const libzcash::SaplingPaymentAddress& a : addrs) {
|
||||
s.insert(a);
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
void ExpectSameEntries(const std::vector<SaplingNoteEntry>& expected,
|
||||
const std::vector<SaplingNoteEntry>& actual)
|
||||
{
|
||||
ASSERT_EQ(expected.size(), actual.size());
|
||||
for (size_t i = 0; i < expected.size(); i++) {
|
||||
SCOPED_TRACE(testing::Message() << "entry " << i);
|
||||
const SaplingNoteEntry& e = expected[i];
|
||||
const SaplingNoteEntry& a = actual[i];
|
||||
EXPECT_EQ(e.op.hash, a.op.hash);
|
||||
EXPECT_EQ(e.op.n, a.op.n);
|
||||
EXPECT_TRUE(e.address == a.address);
|
||||
EXPECT_TRUE(e.note.d == a.note.d);
|
||||
EXPECT_EQ(e.note.pk_d, a.note.pk_d);
|
||||
EXPECT_EQ(e.note.value(), a.note.value());
|
||||
EXPECT_EQ(e.note.r, a.note.r);
|
||||
EXPECT_TRUE(e.memo == a.memo);
|
||||
EXPECT_EQ(e.confirmations, a.confirmations);
|
||||
}
|
||||
}
|
||||
|
||||
// Runs the original and the optimised GetFilteredNotes with the same arguments and
|
||||
// expects the same entries in the same order. Returns the number of entries.
|
||||
size_t ExpectMatchesReference(CWallet& wallet, NamedFilter& filter,
|
||||
int minDepth, int maxDepth,
|
||||
bool ignoreSpent, bool requireSpendingKey, bool ignoreLocked)
|
||||
{
|
||||
SCOPED_TRACE(testing::Message() << "filter=" << filter.name
|
||||
<< " minDepth=" << minDepth << " maxDepth=" << maxDepth
|
||||
<< " ignoreSpent=" << ignoreSpent
|
||||
<< " requireSpendingKey=" << requireSpendingKey
|
||||
<< " ignoreLocked=" << ignoreLocked);
|
||||
|
||||
std::vector<SaplingNoteEntry> expected;
|
||||
std::vector<SaplingNoteEntry> actual;
|
||||
RefGetFilteredNotes(wallet, expected, filter.addrs,
|
||||
minDepth, maxDepth, ignoreSpent, requireSpendingKey, ignoreLocked);
|
||||
wallet.GetFilteredNotes(actual, filter.addrs,
|
||||
minDepth, maxDepth, ignoreSpent, requireSpendingKey, ignoreLocked);
|
||||
ExpectSameEntries(expected, actual);
|
||||
return expected.size();
|
||||
}
|
||||
|
||||
// The reference's result size with maxDepth INT_MAX.
|
||||
size_t RefCount(CWallet& wallet, std::set<libzcash::PaymentAddress> filter, int minDepth,
|
||||
bool ignoreSpent, bool requireSpendingKey, bool ignoreLocked)
|
||||
{
|
||||
std::vector<SaplingNoteEntry> entries;
|
||||
RefGetFilteredNotes(wallet, entries, filter,
|
||||
minDepth, INT_MAX, ignoreSpent, requireSpendingKey, ignoreLocked);
|
||||
return entries.size();
|
||||
}
|
||||
|
||||
// Every filter shape the pre-filter distinguishes.
|
||||
std::vector<NamedFilter> AllFilters(const TestKeys& k)
|
||||
{
|
||||
// 16 recorded, verifiable addresses: the pre-filter is on (the cap is inclusive).
|
||||
std::vector<libzcash::SaplingPaymentAddress> sixteen = {k.a1, k.d1, k.a2, k.a3, k.a4};
|
||||
sixteen.insert(sixteen.end(), k.empty.begin(), k.empty.begin() + 11);
|
||||
// 17: the pre-filter is off only because of the cap.
|
||||
std::vector<libzcash::SaplingPaymentAddress> seventeen = sixteen;
|
||||
seventeen.push_back(k.empty[11]);
|
||||
|
||||
std::set<libzcash::PaymentAddress> invalidOnly;
|
||||
invalidOnly.insert(libzcash::InvalidEncoding());
|
||||
std::set<libzcash::PaymentAddress> invalidAndA2 = invalidOnly;
|
||||
invalidAndA2.insert(k.a2);
|
||||
|
||||
std::vector<NamedFilter> f;
|
||||
f.push_back(NamedFilter{"{}", Filter({})});
|
||||
f.push_back(NamedFilter{"{a1}", Filter({k.a1})});
|
||||
f.push_back(NamedFilter{"{d1}", Filter({k.d1})});
|
||||
f.push_back(NamedFilter{"{a2}", Filter({k.a2})});
|
||||
f.push_back(NamedFilter{"{a3 ivk-only}", Filter({k.a3})});
|
||||
f.push_back(NamedFilter{"{a4 fvk-only}", Filter({k.a4})});
|
||||
f.push_back(NamedFilter{"{a1,a2,a3,a4,d1}", Filter({k.a1, k.a2, k.a3, k.a4, k.d1})});
|
||||
f.push_back(NamedFilter{"{u1 unrecorded}", Filter({k.u1})});
|
||||
f.push_back(NamedFilter{"{a1,u1 unrecorded}", Filter({k.a1, k.u1})});
|
||||
f.push_back(NamedFilter{"{x1 mismatched}", Filter({k.x1})});
|
||||
f.push_back(NamedFilter{"{x1 mismatched,a2}", Filter({k.x1, k.a2})});
|
||||
f.push_back(NamedFilter{"{y1 aliased}", Filter({k.y1})});
|
||||
f.push_back(NamedFilter{"{aN non-canonical}", Filter({k.aN})});
|
||||
f.push_back(NamedFilter{"{no notes}", Filter({k.empty[0]})});
|
||||
f.push_back(NamedFilter{"{foreign}", Filter({k.foreign})});
|
||||
f.push_back(NamedFilter{"{invalid}", invalidOnly});
|
||||
f.push_back(NamedFilter{"{invalid,a2}", invalidAndA2});
|
||||
f.push_back(NamedFilter{"16 addresses", Filter(sixteen)});
|
||||
f.push_back(NamedFilter{"17 addresses", Filter(seventeen)});
|
||||
return f;
|
||||
}
|
||||
|
||||
// decrypt_with_pk_d must succeed exactly when decrypt() does, and must leave pk_d_out
|
||||
// alone when it fails. Returns whether decryption succeeded.
|
||||
bool DecryptBothWays(const libzcash::SaplingEncCiphertext& ct, const uint256& ivk,
|
||||
const uint256& epk, const uint256& cmu)
|
||||
{
|
||||
uint256 sentinel;
|
||||
std::fill(sentinel.begin(), sentinel.end(), 0xa5);
|
||||
uint256 pk_d = sentinel;
|
||||
auto expected = libzcash::SaplingNotePlaintext::decrypt(ct, ivk, epk, cmu);
|
||||
auto actual = libzcash::SaplingNotePlaintext::decrypt_with_pk_d(ct, ivk, epk, cmu, pk_d);
|
||||
EXPECT_EQ(static_cast<bool>(expected), static_cast<bool>(actual));
|
||||
if (!actual) {
|
||||
EXPECT_EQ(pk_d, sentinel);
|
||||
}
|
||||
return static_cast<bool>(actual);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
class WalletScanNotes : public ::testing::Test {
|
||||
protected:
|
||||
boost::filesystem::path datadir;
|
||||
bool hadDatadirArg;
|
||||
std::string oldDatadirArg;
|
||||
bool oldPrintToDebugLog;
|
||||
CBlockIndex* oldTip;
|
||||
std::vector<std::unique_ptr<CBlockIndex>> blocks;
|
||||
std::vector<CTransaction> mempoolTxs;
|
||||
|
||||
void SetUp() override
|
||||
{
|
||||
SelectParams(CBaseChainParams::REGTEST);
|
||||
|
||||
// GetFilteredNotes logs on every call: keep that away from any real datadir and
|
||||
// out of debug.log.
|
||||
hadDatadirArg = mapArgs.count("-datadir") > 0;
|
||||
if (hadDatadirArg) {
|
||||
oldDatadirArg = mapArgs["-datadir"];
|
||||
}
|
||||
datadir = boost::filesystem::temp_directory_path() / boost::filesystem::unique_path();
|
||||
boost::filesystem::create_directories(datadir);
|
||||
mapArgs["-datadir"] = datadir.string();
|
||||
ClearDatadirCache();
|
||||
oldPrintToDebugLog = fPrintToDebugLog;
|
||||
fPrintToDebugLog = false;
|
||||
|
||||
// A fake active chain. Only what CMerkleTx::GetDepthInMainChain reads is set: the
|
||||
// mapBlockIndex entry, phashBlock, the height and pprev.
|
||||
LOCK(cs_main);
|
||||
oldTip = chainActive.Tip();
|
||||
for (int h = 0; h < CHAIN_LENGTH; h++) {
|
||||
std::unique_ptr<CBlockIndex> pindex(new CBlockIndex());
|
||||
pindex->SetHeight(h);
|
||||
pindex->pprev = blocks.empty() ? NULL : blocks.back().get();
|
||||
uint256 hash;
|
||||
hash.begin()[0] = h;
|
||||
hash.begin()[30] = 0x5c;
|
||||
hash.begin()[31] = 0xa9;
|
||||
auto ret = mapBlockIndex.insert(std::make_pair(hash, pindex.get()));
|
||||
ASSERT_TRUE(ret.second);
|
||||
pindex->phashBlock = &ret.first->first;
|
||||
blocks.push_back(std::move(pindex));
|
||||
}
|
||||
chainActive.SetTip(blocks.back().get());
|
||||
}
|
||||
|
||||
void TearDown() override
|
||||
{
|
||||
{
|
||||
LOCK(cs_main);
|
||||
for (const CTransaction& tx : mempoolTxs) {
|
||||
std::list<CTransaction> removed;
|
||||
mempool.remove(tx, removed, false);
|
||||
}
|
||||
mempoolTxs.clear();
|
||||
chainActive.SetTip(oldTip);
|
||||
for (const std::unique_ptr<CBlockIndex>& pindex : blocks) {
|
||||
mapBlockIndex.erase(pindex->GetBlockHash());
|
||||
}
|
||||
blocks.clear();
|
||||
}
|
||||
|
||||
fPrintToDebugLog = oldPrintToDebugLog;
|
||||
if (hadDatadirArg) {
|
||||
mapArgs["-datadir"] = oldDatadirArg;
|
||||
} else {
|
||||
mapArgs.erase("-datadir");
|
||||
}
|
||||
ClearDatadirCache();
|
||||
boost::system::error_code ec;
|
||||
boost::filesystem::remove_all(datadir, ec);
|
||||
|
||||
// The network every other test in this binary selects for itself.
|
||||
SelectParams(CBaseChainParams::MAIN);
|
||||
}
|
||||
|
||||
// Adds a fabricated tx to the wallet the way LoadWallet does: AddToWallet(wtx, true,
|
||||
// NULL) skips the database and proof checks but fills the nullifier and spend maps.
|
||||
// `owned` become outputs 0..n-1 with wallet note data, `foreign` outputs follow with
|
||||
// none, and `spends` are the nullifiers the tx reveals. depth >= 1 puts the tx in the
|
||||
// fake chain, 0 in the mempool, -1 nowhere (like a conflicted tx).
|
||||
uint256 AddTx(CWallet& wallet, int depth,
|
||||
const std::vector<OwnedOutput>& owned,
|
||||
const std::vector<libzcash::SaplingPaymentAddress>& foreign,
|
||||
const std::vector<uint256>& spends,
|
||||
bool transparentOutput = false)
|
||||
{
|
||||
assert(depth >= -1 && depth <= CHAIN_LENGTH);
|
||||
|
||||
CMutableTransaction mtx;
|
||||
mtx.fOverwintered = true;
|
||||
mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
|
||||
mtx.nVersion = SAPLING_TX_VERSION;
|
||||
for (const OwnedOutput& o : owned) {
|
||||
mtx.vShieldedOutput.push_back(MakeOutput(o.addr, o.value));
|
||||
}
|
||||
for (const libzcash::SaplingPaymentAddress& addr : foreign) {
|
||||
mtx.vShieldedOutput.push_back(MakeOutput(addr, 1));
|
||||
}
|
||||
for (const uint256& nf : spends) {
|
||||
SpendDescription sd;
|
||||
sd.nullifier = nf;
|
||||
sd.zkproof.fill(0);
|
||||
sd.spendAuthSig.fill(0);
|
||||
mtx.vShieldedSpend.push_back(sd);
|
||||
}
|
||||
if (transparentOutput) {
|
||||
mtx.vout.push_back(CTxOut(1000, CScript() << OP_TRUE));
|
||||
}
|
||||
CTransaction tx(mtx);
|
||||
|
||||
CWalletTx wtx(&wallet, tx);
|
||||
for (size_t i = 0; i < owned.size(); i++) {
|
||||
SaplingNoteData nd(owned[i].ivk);
|
||||
nd.nullifier = owned[i].nullifier;
|
||||
wtx.mapSaplingNoteData[SaplingOutPoint(tx.GetHash(), i)] = nd;
|
||||
}
|
||||
if (depth >= 1) {
|
||||
wtx.hashBlock = blocks[CHAIN_LENGTH - depth]->GetBlockHash();
|
||||
wtx.nIndex = 0;
|
||||
// There is no real block to check a merkle branch against.
|
||||
wtx.fMerkleVerified = true;
|
||||
} else if (depth == 0) {
|
||||
mempool.addUnchecked(tx.GetHash(),
|
||||
CTxMemPoolEntry(tx, 0, GetTime(), 0.0, chainActive.Height(), true, false, SAPLING_BRANCH_ID),
|
||||
false);
|
||||
mempoolTxs.push_back(tx);
|
||||
}
|
||||
wallet.AddToWallet(wtx, true, NULL);
|
||||
return tx.GetHash();
|
||||
}
|
||||
|
||||
// The test wallet. Every key state, every recorded-address state and every note state
|
||||
// GetFilteredNotes distinguishes, spread over several depths:
|
||||
// K1 spending key: a1 (default), d1 (recorded diversified), u1 (unrecorded),
|
||||
// x1 (recorded with K2's ivk, as z_importviewingkey allows), y1 (recorded with
|
||||
// ivk1 + r_J, which still generates y1);
|
||||
// K2 spending key: a2; K3 ivk only: a3; K4 fvk only: a4;
|
||||
// ivkN: a non-canonical ivk in [2^251, r_J) and its address aN;
|
||||
// 12 more spending keys with no notes; one foreign key.
|
||||
// Notes are spent in a block, spent in the mempool, "spent" by a conflicted tx (so
|
||||
// still unspent), unspent, without a cached nullifier although spent on chain, and
|
||||
// locked. There are also txs without note data (spends only, transparent only, and
|
||||
// a shielded output to a foreign address).
|
||||
void BuildWallet(CWallet& wallet, TestKeys& k)
|
||||
{
|
||||
LOCK2(cs_main, wallet.cs_wallet);
|
||||
|
||||
libzcash::SaplingExtendedSpendingKey xsk1 = KeyFromSeedByte(0x11);
|
||||
libzcash::SaplingExtendedSpendingKey xsk2 = KeyFromSeedByte(0x22);
|
||||
libzcash::SaplingExtendedSpendingKey xsk3 = KeyFromSeedByte(0x33);
|
||||
libzcash::SaplingExtendedSpendingKey xsk4 = KeyFromSeedByte(0x44);
|
||||
libzcash::SaplingExtendedSpendingKey xskN = KeyFromSeedByte(0x55);
|
||||
|
||||
k.fvk1 = xsk1.expsk.full_viewing_key();
|
||||
k.fvk2 = xsk2.expsk.full_viewing_key();
|
||||
libzcash::SaplingFullViewingKey fvk4 = xsk4.expsk.full_viewing_key();
|
||||
k.ivk1 = k.fvk1.in_viewing_key();
|
||||
k.ivk2 = k.fvk2.in_viewing_key();
|
||||
k.ivk3 = xsk3.expsk.full_viewing_key().in_viewing_key();
|
||||
k.ivk4 = fvk4.in_viewing_key();
|
||||
|
||||
k.a1 = xsk1.DefaultAddress();
|
||||
k.d1 = DiversifiedAddress(xsk1, 50);
|
||||
k.u1 = DiversifiedAddress(xsk1, 100);
|
||||
k.x1 = DiversifiedAddress(xsk1, 150);
|
||||
k.y1 = DiversifiedAddress(xsk1, 200);
|
||||
std::set<libzcash::SaplingPaymentAddress> k1addrs = {k.a1, k.d1, k.u1, k.x1, k.y1};
|
||||
ASSERT_EQ(k1addrs.size(), (size_t)5);
|
||||
k.a2 = xsk2.DefaultAddress();
|
||||
k.a3 = xsk3.DefaultAddress();
|
||||
k.a4 = xsk4.DefaultAddress();
|
||||
k.foreign = KeyFromSeedByte(0x66).DefaultAddress();
|
||||
|
||||
// Genuine ivks are canonical (top 5 bits clear, so below 2^251).
|
||||
ASSERT_EQ(k.ivk1.begin()[31] & 0xF8, 0);
|
||||
ASSERT_EQ(k.ivk2.begin()[31] & 0xF8, 0);
|
||||
|
||||
// ivk1 + r_J generates exactly ivk1's addresses. If this ever stops holding, the
|
||||
// y1 case no longer exercises the pre-filter's canonical-ivk guard.
|
||||
k.aliasIvk1 = libzcash::SaplingIncomingViewingKey(ArithToUint256(
|
||||
UintToArith256(k.ivk1) + UintToArith256(uint256S(JUBJUB_R_HEX))));
|
||||
ASSERT_NE(k.aliasIvk1.begin()[31] & 0xF8, 0);
|
||||
auto aliasY1 = k.aliasIvk1.address(k.y1.d);
|
||||
ASSERT_TRUE(static_cast<bool>(aliasY1));
|
||||
EXPECT_TRUE(aliasY1.get() == k.y1);
|
||||
|
||||
// ivkN in [2^251, 2^251 + 2^248), below r_J: ka_agree accepts it, so notes to aN
|
||||
// decrypt normally, but it is not canonical.
|
||||
uint256 rawN = xskN.expsk.full_viewing_key().in_viewing_key();
|
||||
rawN.begin()[31] = 0x08;
|
||||
k.ivkN = libzcash::SaplingIncomingViewingKey(rawN);
|
||||
k.aN = k.ivkN.address(k.a1.d).get();
|
||||
|
||||
ASSERT_TRUE(wallet.AddSaplingZKey(xsk1, k.a1));
|
||||
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(k.ivk1, k.d1));
|
||||
ASSERT_TRUE(wallet.AddSaplingZKey(xsk2, k.a2));
|
||||
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(k.ivk3, k.a3));
|
||||
ASSERT_TRUE(wallet.AddSaplingFullViewingKey(fvk4, k.a4));
|
||||
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(k.ivk2, k.x1));
|
||||
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(k.aliasIvk1, k.y1));
|
||||
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(k.ivkN, k.aN));
|
||||
for (int i = 0; i < 12; i++) {
|
||||
libzcash::SaplingExtendedSpendingKey xsk = KeyFromSeedByte(0x70 + i);
|
||||
libzcash::SaplingPaymentAddress addr = xsk.DefaultAddress();
|
||||
ASSERT_TRUE(wallet.AddSaplingZKey(xsk, addr));
|
||||
k.empty.push_back(addr);
|
||||
}
|
||||
|
||||
const boost::optional<uint256> noNf;
|
||||
std::vector<uint256> noSpends;
|
||||
std::vector<libzcash::SaplingPaymentAddress> noForeign;
|
||||
|
||||
// Mixed ownership in one tx; Nf(2) is spent in a block.
|
||||
AddTx(wallet, 5, {{k.a1, k.ivk1, 1001, Nf(1)},
|
||||
{k.a1, k.ivk1, 1002, Nf(2)},
|
||||
{k.a2, k.ivk2, 2001, Nf(3)}}, noForeign, noSpends);
|
||||
// Nf(4) is spent in the mempool. The IVK-only note has no cached nullifier, but its
|
||||
// on-chain nullifier Nf(5) is spent in a block: it must still count as unspent.
|
||||
AddTx(wallet, 1, {{k.d1, k.ivk1, 1003, Nf(4)},
|
||||
{k.a3, k.ivk3, 3001, noNf}}, noForeign, noSpends);
|
||||
// u1's note is locked; x1's note is "spent" only by a conflicted tx.
|
||||
uint256 hashUX = AddTx(wallet, 3, {{k.u1, k.ivk1, 1004, Nf(6)},
|
||||
{k.x1, k.ivk1, 1005, Nf(7)}}, noForeign, noSpends);
|
||||
// In the mempool (depth 0).
|
||||
AddTx(wallet, 0, {{k.a1, k.ivk1, 1006, Nf(8)},
|
||||
{k.a4, k.ivk4, 4001, Nf(9)}}, noForeign, noSpends);
|
||||
// Conflicted (depth -1).
|
||||
AddTx(wallet, -1, {{k.a2, k.ivk2, 2002, Nf(10)}}, noForeign, noSpends);
|
||||
// Deepest block; Nf(11) is spent in a block.
|
||||
AddTx(wallet, 10, {{k.y1, k.ivk1, 1007, Nf(11)},
|
||||
{k.aN, k.ivkN, 5001, noNf}}, noForeign, noSpends);
|
||||
// a1's note is locked; one output goes to a foreign address (no note data).
|
||||
uint256 hashLocked = AddTx(wallet, 2, {{k.a1, k.ivk1, 1008, Nf(12)},
|
||||
{k.a2, k.ivk2, 2003, Nf(13)}},
|
||||
{k.foreign}, noSpends);
|
||||
// Nf(14) is spent in a block.
|
||||
AddTx(wallet, 4, {{k.a3, k.ivk3, 3002, noNf},
|
||||
{k.aN, k.ivkN, 5002, Nf(14)},
|
||||
{k.x1, k.ivk1, 1009, Nf(15)}}, noForeign, noSpends);
|
||||
AddTx(wallet, 1, {{k.d1, k.ivk1, 1010, Nf(16)}}, noForeign, noSpends);
|
||||
k.nNotes = 18;
|
||||
|
||||
// Txs without note data: the spenders (in a block, in the mempool, conflicted), a
|
||||
// transparent-only tx and a tx whose only output pays a foreign address.
|
||||
AddTx(wallet, 2, {}, noForeign, {Nf(2), Nf(5), Nf(11), Nf(14)});
|
||||
AddTx(wallet, 0, {}, noForeign, {Nf(4)});
|
||||
AddTx(wallet, -1, {}, noForeign, {Nf(7)});
|
||||
AddTx(wallet, 3, {}, noForeign, noSpends, true);
|
||||
AddTx(wallet, 1, {}, {k.foreign}, noSpends);
|
||||
|
||||
wallet.LockNote(SaplingOutPoint(hashUX, 0));
|
||||
wallet.LockNote(SaplingOutPoint(hashLocked, 0));
|
||||
}
|
||||
};
|
||||
|
||||
// The fixture really has what the equivalence tests rely on: every note is visible, the
|
||||
// spent, depth, key and lock states are the intended ones, and the regression cases
|
||||
// (unrecorded, mismatched, aliased and non-canonical addresses) hold notes, so equivalence
|
||||
// on them is not vacuous.
|
||||
TEST_F(WalletScanNotes, ReferenceSeesTheFixture)
|
||||
{
|
||||
CWallet wallet;
|
||||
TestKeys k;
|
||||
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
|
||||
|
||||
std::set<libzcash::PaymentAddress> none;
|
||||
std::vector<SaplingNoteEntry> all;
|
||||
RefGetFilteredNotes(wallet, all, none, -1, INT_MAX, false, false, false);
|
||||
EXPECT_EQ(all.size(), k.nNotes);
|
||||
std::set<int> depths;
|
||||
for (const SaplingNoteEntry& e : all) {
|
||||
depths.insert(e.confirmations);
|
||||
}
|
||||
EXPECT_EQ(depths, (std::set<int>{-1, 0, 1, 2, 3, 4, 5, 10}));
|
||||
|
||||
// Spent: Nf(2), Nf(11) and Nf(14) in a block, Nf(4) in the mempool. Not spent: Nf(7)
|
||||
// (conflicted spender) and the IVK-only note without a cached nullifier.
|
||||
EXPECT_EQ(RefCount(wallet, none, -1, true, false, false), k.nNotes - 4);
|
||||
// minDepth 0 drops the conflicted tx (1 note), minDepth 1 also the mempool tx (2).
|
||||
EXPECT_EQ(RefCount(wallet, none, 0, false, false, false), k.nNotes - 1);
|
||||
EXPECT_EQ(RefCount(wallet, none, 1, false, false, false), k.nNotes - 3);
|
||||
EXPECT_EQ(RefCount(wallet, none, -1, false, false, true), k.nNotes - 2);
|
||||
// Spending key reachable through the recorded address: a1 (4), d1 (2), a2 (3), and
|
||||
// x1 (2), because x1 is recorded with K2's ivk and the wallet has K2's spending key.
|
||||
EXPECT_EQ(RefCount(wallet, none, -1, false, true, false), (size_t)11);
|
||||
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.a1}), -1, false, false, false), (size_t)4);
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.d1}), -1, false, false, false), (size_t)2);
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.u1}), -1, false, false, false), (size_t)1);
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.x1}), -1, false, false, false), (size_t)2);
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.y1}), -1, false, false, false), (size_t)1);
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.a2}), -1, false, false, false), (size_t)3);
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.a3}), -1, false, false, false), (size_t)2);
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.a4}), -1, false, false, false), (size_t)1);
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.aN}), -1, false, false, false), (size_t)2);
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.empty[0]}), -1, false, false, false), (size_t)0);
|
||||
EXPECT_EQ(RefCount(wallet, Filter({k.foreign}), -1, false, false, false), (size_t)0);
|
||||
}
|
||||
|
||||
// Every filter shape x ignoreSpent x requireSpendingKey x ignoreLocked x minDepth {0, 1}.
|
||||
TEST_F(WalletScanNotes, MatchesReferenceAcrossFilters)
|
||||
{
|
||||
CWallet wallet;
|
||||
TestKeys k;
|
||||
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
|
||||
|
||||
std::vector<NamedFilter> filters = AllFilters(k);
|
||||
size_t total = 0;
|
||||
for (NamedFilter& f : filters) {
|
||||
for (int minDepth : {0, 1}) {
|
||||
for (bool ignoreSpent : {false, true}) {
|
||||
for (bool requireSpendingKey : {false, true}) {
|
||||
for (bool ignoreLocked : {false, true}) {
|
||||
total += ExpectMatchesReference(wallet, f, minDepth, INT_MAX,
|
||||
ignoreSpent, requireSpendingKey, ignoreLocked);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EXPECT_GT(total, (size_t)0);
|
||||
}
|
||||
|
||||
// The depth window: minDepth {-1, 0, 1} x maxDepth {0, 1, 3, INT_MAX} x the three flags,
|
||||
// unfiltered and with a single (pre-filtered) address.
|
||||
TEST_F(WalletScanNotes, MatchesReferenceAcrossDepths)
|
||||
{
|
||||
CWallet wallet;
|
||||
TestKeys k;
|
||||
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
|
||||
|
||||
std::vector<NamedFilter> filters;
|
||||
filters.push_back(NamedFilter{"{}", Filter({})});
|
||||
filters.push_back(NamedFilter{"{a1}", Filter({k.a1})});
|
||||
size_t total = 0;
|
||||
for (NamedFilter& f : filters) {
|
||||
for (int minDepth : {-1, 0, 1}) {
|
||||
for (int maxDepth : {0, 1, 3, INT_MAX}) {
|
||||
for (bool ignoreSpent : {false, true}) {
|
||||
for (bool requireSpendingKey : {false, true}) {
|
||||
for (bool ignoreLocked : {false, true}) {
|
||||
total += ExpectMatchesReference(wallet, f, minDepth, maxDepth,
|
||||
ignoreSpent, requireSpendingKey, ignoreLocked);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EXPECT_GT(total, (size_t)0);
|
||||
}
|
||||
|
||||
// The single-address string overload (z_getbalance and friends): it decodes the address
|
||||
// and calls the set overload with maxDepth INT_MAX and ignoreLocked true.
|
||||
TEST_F(WalletScanNotes, MatchesReferenceForAddressString)
|
||||
{
|
||||
CWallet wallet;
|
||||
TestKeys k;
|
||||
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
|
||||
|
||||
std::vector<std::string> addresses = {
|
||||
"",
|
||||
EncodePaymentAddress(k.a1),
|
||||
EncodePaymentAddress(k.u1),
|
||||
EncodePaymentAddress(k.x1),
|
||||
EncodePaymentAddress(k.foreign),
|
||||
"not-an-address",
|
||||
};
|
||||
size_t total = 0;
|
||||
for (const std::string& address : addresses) {
|
||||
for (int minDepth : {0, 1}) {
|
||||
for (bool ignoreSpent : {false, true}) {
|
||||
for (bool requireSpendingKey : {false, true}) {
|
||||
SCOPED_TRACE(testing::Message() << "address=\"" << address << "\""
|
||||
<< " minDepth=" << minDepth << " ignoreSpent=" << ignoreSpent
|
||||
<< " requireSpendingKey=" << requireSpendingKey);
|
||||
std::set<libzcash::PaymentAddress> filter;
|
||||
if (address.length() > 0) {
|
||||
filter.insert(DecodePaymentAddress(address));
|
||||
}
|
||||
std::vector<SaplingNoteEntry> expected;
|
||||
std::vector<SaplingNoteEntry> actual;
|
||||
RefGetFilteredNotes(wallet, expected, filter,
|
||||
minDepth, INT_MAX, ignoreSpent, requireSpendingKey, true);
|
||||
wallet.GetFilteredNotes(actual, address, minDepth, ignoreSpent, requireSpendingKey);
|
||||
ExpectSameEntries(expected, actual);
|
||||
total += expected.size();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EXPECT_GT(total, (size_t)0);
|
||||
}
|
||||
|
||||
// An encrypted, locked wallet: crypted spending keys still satisfy requireSpendingKey and
|
||||
// the ivk/fvk maps are not encrypted, so both implementations must still agree.
|
||||
TEST_F(WalletScanNotes, MatchesReferenceWhenEncryptedAndLocked)
|
||||
{
|
||||
EncryptableWallet wallet;
|
||||
TestKeys k;
|
||||
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
|
||||
|
||||
CKeyingMaterial masterKey(WALLET_CRYPTO_KEY_SIZE, 0x5a);
|
||||
ASSERT_TRUE(wallet.EncryptKeysForTest(masterKey));
|
||||
ASSERT_TRUE(wallet.Lock());
|
||||
ASSERT_TRUE(wallet.IsCrypted());
|
||||
ASSERT_TRUE(wallet.IsLocked());
|
||||
EXPECT_TRUE(wallet.HaveSaplingSpendingKey(k.fvk1));
|
||||
|
||||
std::vector<NamedFilter> filters = AllFilters(k);
|
||||
size_t total = 0;
|
||||
size_t spendable = 0;
|
||||
for (NamedFilter& f : filters) {
|
||||
for (bool ignoreSpent : {false, true}) {
|
||||
for (bool requireSpendingKey : {false, true}) {
|
||||
for (bool ignoreLocked : {false, true}) {
|
||||
size_t n = ExpectMatchesReference(wallet, f, 0, INT_MAX,
|
||||
ignoreSpent, requireSpendingKey, ignoreLocked);
|
||||
total += n;
|
||||
if (requireSpendingKey) {
|
||||
spendable += n;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
EXPECT_GT(total, (size_t)0);
|
||||
EXPECT_GT(spendable, (size_t)0);
|
||||
}
|
||||
|
||||
// decrypt_with_pk_d returns the same plaintext as decrypt() and the pk_d that
|
||||
// ivk.address(d) recomputes, and fails exactly when decrypt() fails.
|
||||
TEST_F(WalletScanNotes, DecryptWithPkDMatchesDecrypt)
|
||||
{
|
||||
CWallet wallet;
|
||||
TestKeys k;
|
||||
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
|
||||
|
||||
LOCK2(cs_main, wallet.cs_wallet);
|
||||
size_t checked = 0;
|
||||
for (const auto& p : wallet.mapWallet) {
|
||||
const CWalletTx& wtx = p.second;
|
||||
for (const auto& item : wtx.mapSaplingNoteData) {
|
||||
SCOPED_TRACE(testing::Message() << "note " << item.first.ToString());
|
||||
const OutputDescription& od = wtx.vShieldedOutput[item.first.n];
|
||||
const libzcash::SaplingIncomingViewingKey& ivk = item.second.ivk;
|
||||
|
||||
uint256 pk_d;
|
||||
auto expected = libzcash::SaplingNotePlaintext::decrypt(
|
||||
od.encCiphertext, ivk, od.ephemeralKey, od.cm);
|
||||
auto actual = libzcash::SaplingNotePlaintext::decrypt_with_pk_d(
|
||||
od.encCiphertext, ivk, od.ephemeralKey, od.cm, pk_d);
|
||||
ASSERT_TRUE(static_cast<bool>(expected));
|
||||
ASSERT_TRUE(static_cast<bool>(actual));
|
||||
EXPECT_TRUE(actual->d == expected->d);
|
||||
EXPECT_EQ(actual->value(), expected->value());
|
||||
EXPECT_EQ(actual->rcm, expected->rcm);
|
||||
EXPECT_TRUE(actual->memo() == expected->memo());
|
||||
auto addr = ivk.address(actual->d);
|
||||
ASSERT_TRUE(static_cast<bool>(addr));
|
||||
EXPECT_EQ(pk_d, addr->pk_d);
|
||||
checked++;
|
||||
|
||||
// Wrong ivk: the AEAD fails.
|
||||
const libzcash::SaplingIncomingViewingKey& wrongIvk = (ivk == k.ivk2) ? k.ivk1 : k.ivk2;
|
||||
EXPECT_FALSE(DecryptBothWays(od.encCiphertext, wrongIvk, od.ephemeralKey, od.cm));
|
||||
// Wrong cmu: decryption and ivk_to_pkd succeed, the commitment check fails, so
|
||||
// pk_d has been computed but must not reach pk_d_out.
|
||||
uint256 badCm = od.cm;
|
||||
badCm.begin()[0] ^= 1;
|
||||
EXPECT_FALSE(DecryptBothWays(od.encCiphertext, ivk, od.ephemeralKey, badCm));
|
||||
// Tampered ciphertext.
|
||||
libzcash::SaplingEncCiphertext badCt = od.encCiphertext;
|
||||
badCt[0] ^= 1;
|
||||
EXPECT_FALSE(DecryptBothWays(badCt, ivk, od.ephemeralKey, od.cm));
|
||||
// ivk1 + r_J: whatever librustzcash decides for a non-canonical scalar, both agree.
|
||||
if (ivk == k.ivk1) {
|
||||
DecryptBothWays(od.encCiphertext, k.aliasIvk1, od.ephemeralKey, od.cm);
|
||||
}
|
||||
}
|
||||
}
|
||||
EXPECT_EQ(checked, k.nNotes);
|
||||
}
|
||||
+1
-58
@@ -513,10 +513,6 @@ std::string HelpMessage(HelpMessageMode mode)
|
||||
// If this is used incorrectly (-rescanheight too large), then the local wallet may attempt to spend funds which it does not have witness data about
|
||||
// which will cause a "missing inputs" error when added to the mempool. Rescanning from correct height will fix this.
|
||||
strUsage += HelpMessageOpt("-keepnotewitnesscache", _("Keep partial Sapling Note Witness cache. Must be used with -rescanheight to find missing cache items."));
|
||||
if (showDebug)
|
||||
strUsage += HelpMessageOpt("-notecachewarmup", "Decrypt all wallet notes into memory at startup, so the first balance call after a restart doesn't (default: 1)");
|
||||
if (showDebug)
|
||||
strUsage += HelpMessageOpt("-checknotecache", "On every balance or note-listing call, decrypt each cached note it considers again and log and repair any difference from the cache (slow; default: 0)");
|
||||
strUsage += HelpMessageOpt("-rescan", _("Rescan the block chain for missing wallet transactions") + " " + _("on startup"));
|
||||
strUsage += HelpMessageOpt("-rescanheight", _("Rescan from specified height when rescan=1 on startup"));
|
||||
strUsage += HelpMessageOpt("-salvagewallet", _("Attempt to recover private keys from a corrupt wallet.dat") + " " + _("on startup"));
|
||||
@@ -1020,8 +1016,7 @@ extern int32_t HUSH_REWIND;
|
||||
|
||||
// --- Adaptive coins-cache sizing -------------------------------------------------------------
|
||||
// The in-memory UTXO/coins cache (nCoinCacheUsage) is the biggest lever on IBD speed: a bigger
|
||||
// cache means fewer size-triggered chainstate flushes to disk (the chainstate is also flushed at
|
||||
// least every DATABASE_FLUSH_INTERVAL / DATABASE_FLUSH_BLOCKS, see main.h). We size it to use most of RAM, but a scheduled
|
||||
// cache means far fewer chainstate flushes to disk. We size it to use most of RAM, but a scheduled
|
||||
// background task (AdjustCoinCacheForMemoryPressure, registered in AppInit2) shrinks the target when
|
||||
// free system memory runs low — e.g. the user opens other apps — and grows it back when memory frees
|
||||
// up, always leaving a reserve free for the rest of the system. The existing per-block flush
|
||||
@@ -2234,9 +2229,6 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
|
||||
|
||||
// ********************************************************* Step 8: load wallet
|
||||
#ifdef ENABLE_WALLET
|
||||
// Set when the startup rescan below is interrupted: the wallet's state then ends at the scan's
|
||||
// own checkpoint, not at the tip the notifier starts from.
|
||||
bool fWalletScanIncomplete = false;
|
||||
if (fDisableWallet) {
|
||||
pwalletMain = NULL;
|
||||
LogPrintf("Wallet disabled!\n");
|
||||
@@ -2508,14 +2500,6 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
|
||||
}
|
||||
}
|
||||
|
||||
// Sweep and consolidation pick notes with GetFilteredNotes at minDepth > 1. Older versions
|
||||
// located each note's block by re-reading its tx through the txindex, so on a -txindex=0 node
|
||||
// every fully shielded tx was skipped and these never moved those notes. Now they will.
|
||||
if (!fTxIndex && (pwalletMain->fSweepEnabled || pwalletMain->fSaplingConsolidationEnabled)) {
|
||||
LogPrintf("%s: -txindex=0 with -zsweep/-consolidation: fully shielded notes, which versions before "
|
||||
"v1.3.1 never saw on such nodes, are now included in sweeps and consolidations\n", __func__);
|
||||
}
|
||||
|
||||
//Set Automatic Coinbase Shielding (default ON, conditional: self-guards
|
||||
//on nodes where it cannot act - no owned coinbase, external mineraddress,
|
||||
//or locked wallet). Closes the transparent-coinbase leak for miners.
|
||||
@@ -2682,7 +2666,6 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
|
||||
} else {
|
||||
LogPrintf("Rescan did not complete; leaving the best-block locator at the scan's own "
|
||||
"checkpoint so the remaining range is rescanned on the next start\n");
|
||||
fWalletScanIncomplete = true;
|
||||
}
|
||||
|
||||
// Restore wallet transaction metadata after -zapwallettxes=1
|
||||
@@ -2710,15 +2693,6 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
|
||||
}
|
||||
}
|
||||
}
|
||||
// Decrypt the wallet's notes into the in-memory plaintext cache now, while RPC is still in warmup
|
||||
// and before the wallet notification thread (started below) can want cs_wallet. Otherwise the
|
||||
// first balance call after a restart decrypts them all under cs_main.
|
||||
fCheckNoteCache = GetBoolArg("-checknotecache", false);
|
||||
if (GetBoolArg("-notecachewarmup", true) && !ShutdownRequested()) {
|
||||
uiInterface.InitMessage(_("Decrypting wallet notes..."));
|
||||
pwalletMain->WarmSaplingNoteCache();
|
||||
}
|
||||
|
||||
pwalletMain->SetBroadcastTransactions(GetBoolArg("-walletbroadcast", true));
|
||||
} // (!fDisableWallet)
|
||||
#else // ENABLE_WALLET
|
||||
@@ -2762,21 +2736,6 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
|
||||
{
|
||||
LOCK(cs_main);
|
||||
pindexLastTip = chainActive.Tip();
|
||||
#ifdef ENABLE_WALLET
|
||||
// The wallet's state reflects the block the notifier starts from, unless the startup
|
||||
// rescan was interrupted: then it ends at the scan's checkpoint, and the hourly
|
||||
// SetBestChain must keep stamping that so the next start rescans the rest.
|
||||
if (pwalletMain) {
|
||||
const CBlockIndex* pindexWallet = pindexLastTip;
|
||||
if (fWalletScanIncomplete) {
|
||||
CWalletDB walletdb(pwalletMain->strWalletFile);
|
||||
CBlockLocator locator;
|
||||
if (walletdb.ReadBestBlock(locator))
|
||||
pindexWallet = FindForkInGlobalIndex(chainActive, locator);
|
||||
}
|
||||
pwalletMain->SetWalletTip(pindexWallet);
|
||||
}
|
||||
#endif
|
||||
}
|
||||
boost::function<void()> threadnotifywallets = boost::bind(&ThreadNotifyWallets, pindexLastTip);
|
||||
threadGroup.create_thread(
|
||||
@@ -2818,24 +2777,8 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
|
||||
uiInterface.InitMessage(_("Activating best chain..."));
|
||||
// scan for better chains in the block chain database, that are not yet connected in the active best chain
|
||||
CValidationState state;
|
||||
const CBlockIndex* pindexBeforeActivate = NULL;
|
||||
{
|
||||
LOCK(cs_main);
|
||||
pindexBeforeActivate = chainActive.Tip();
|
||||
}
|
||||
if ( !ActivateBestChain(true,state))
|
||||
strErrors << "Failed to connect best block";
|
||||
else {
|
||||
// Persist a startup replay now: the wallet's witness rebuild runs next and can hold
|
||||
// cs_main for a long time, and a node killed before any later flush would replay it all again.
|
||||
bool fTipMoved;
|
||||
{
|
||||
LOCK(cs_main);
|
||||
fTipMoved = chainActive.Tip() != pindexBeforeActivate;
|
||||
}
|
||||
if (fTipMoved)
|
||||
FlushStateToDisk();
|
||||
}
|
||||
} else {
|
||||
LogPrintf("HUSH_REWIND < 0\n");
|
||||
}
|
||||
|
||||
+2
-12
@@ -3738,7 +3738,6 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
|
||||
static int64_t nLastWrite = 0;
|
||||
static int64_t nLastFlush = 0;
|
||||
static int64_t nLastSetChain = 0;
|
||||
static int nLastFlushHeight = -1;
|
||||
std::set<int> setFilesToPrune;
|
||||
bool fFlushForPrune = false;
|
||||
try {
|
||||
@@ -3764,9 +3763,6 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
|
||||
if (nLastSetChain == 0) {
|
||||
nLastSetChain = nNow;
|
||||
}
|
||||
if (nLastFlushHeight < 0) {
|
||||
nLastFlushHeight = chainActive.Height();
|
||||
}
|
||||
size_t cacheSize = pcoinsTip->DynamicMemoryUsage();
|
||||
// The cache is large and close to the limit, but we have time now (not in the middle of a block processing).
|
||||
bool fCacheLarge = mode == FLUSH_STATE_PERIODIC && cacheSize * (10.0/9) > nCoinCacheUsage;
|
||||
@@ -3776,10 +3772,8 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
|
||||
bool fPeriodicWrite = mode == FLUSH_STATE_PERIODIC && nNow > nLastWrite + (int64_t)DATABASE_WRITE_INTERVAL * 1000000;
|
||||
// It's been very long since we flushed the cache. Do this infrequently, to optimize cache usage.
|
||||
bool fPeriodicFlush = mode == FLUSH_STATE_PERIODIC && nNow > nLastFlush + (int64_t)DATABASE_FLUSH_INTERVAL * 1000000;
|
||||
// Many blocks connected since the last flush, e.g. a long replay inside one ActivateBestChain call.
|
||||
bool fBlockCountFlush = (mode == FLUSH_STATE_IF_NEEDED || mode == FLUSH_STATE_PERIODIC) && chainActive.Height() - nLastFlushHeight >= DATABASE_FLUSH_BLOCKS;
|
||||
// Combine all conditions that result in a full cache flush.
|
||||
bool fDoFullFlush = (mode == FLUSH_STATE_ALWAYS) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune || fBlockCountFlush;
|
||||
bool fDoFullFlush = (mode == FLUSH_STATE_ALWAYS) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune;
|
||||
// Write blocks and block index to disk.
|
||||
if (fDoFullFlush || fPeriodicWrite) {
|
||||
// Depend on nMinDiskSpace to ensure we can write block index
|
||||
@@ -3833,7 +3827,6 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
|
||||
if (!pcoinsTip->Flush())
|
||||
return AbortNode(state, "Failed to write to coin database");
|
||||
nLastFlush = nNow;
|
||||
nLastFlushHeight = chainActive.Height();
|
||||
}
|
||||
if ((mode == FLUSH_STATE_ALWAYS || mode == FLUSH_STATE_PERIODIC) && nNow > nLastSetChain + (int64_t)DATABASE_WRITE_INTERVAL * 1000000) {
|
||||
// Update best block in wallet (so we can detect restored wallets).
|
||||
@@ -5922,10 +5915,7 @@ FILE* OpenDiskFile(const CDiskBlockPos &pos, const char *prefix, bool fReadOnly)
|
||||
return NULL;
|
||||
boost::filesystem::path path = GetBlockPosFilename(pos, prefix);
|
||||
boost::filesystem::create_directories(path.parent_path());
|
||||
// Read-only opens get a read-only FILE (every fReadOnly caller only reads). With "rb+", a write through a
|
||||
// reused fd number landed here: when the wolfSSL CloseSocket macro (see net.h) closed the wallet's BDB fd
|
||||
// behind its back, BDB's next page write went to whatever reopened that number -- a block file opened here.
|
||||
FILE* file = fopen(path.string().c_str(), fReadOnly ? "rb" : "rb+");
|
||||
FILE* file = fopen(path.string().c_str(), "rb+");
|
||||
if (!file && !fReadOnly)
|
||||
file = fopen(path.string().c_str(), "wb+");
|
||||
if (!file) {
|
||||
|
||||
+2
-9
@@ -143,15 +143,8 @@ static const unsigned int MAX_LOCATOR_SZ = 101;
|
||||
static unsigned int BLOCK_DOWNLOAD_WINDOW = 1024;
|
||||
/** Time to wait (in seconds) between writing blocks/block index to disk. */
|
||||
static const unsigned int DATABASE_WRITE_INTERVAL = 60 * 60;
|
||||
/** Time to wait (in seconds) between flushing chainstate to disk. Upstream uses 24h to keep a large
|
||||
* coins cache warm, but DragonX's cache stays a few MiB (private chain, few transparent outputs), so
|
||||
* the size triggers never fire and this timer is all that persists the chainstate while a node runs:
|
||||
* a node that never reached 24h uptime or a clean shutdown replayed weeks of blocks on every start. */
|
||||
static const unsigned int DATABASE_FLUSH_INTERVAL = 60 * 60;
|
||||
/** Also flush chainstate once this many blocks were connected since the last flush. A long
|
||||
* ActivateBestChain (startup replay, catching up after downtime) only flushes IF_NEEDED until it
|
||||
* returns, so without this a node killed part-way through loses the whole replay. */
|
||||
static const int DATABASE_FLUSH_BLOCKS = 5000;
|
||||
/** Time to wait (in seconds) between flushing chainstate to disk. */
|
||||
static const unsigned int DATABASE_FLUSH_INTERVAL = 24 * 60 * 60;
|
||||
/** Maximum length of reject messages. */
|
||||
static const unsigned int MAX_REJECT_MESSAGE_LENGTH = 111;
|
||||
static const int64_t DEFAULT_MAX_TIP_AGE = 24 * 60 * 60;
|
||||
|
||||
+10
-7
@@ -45,13 +45,6 @@
|
||||
#include <wolfssl/options.h>
|
||||
#include <wolfssl/ssl.h>
|
||||
#include <hush/tlsmanager.cpp>
|
||||
|
||||
// CloseSocket must be util/sock.cpp's CloseSocket(SOCKET&), which invalidates the handle it closes. If
|
||||
// wolfSSL's `#define CloseSocket(s) close(s)` survived into this file (see net.h), each close here would leave
|
||||
// its stale fd number behind to be closed again after another thread reused it.
|
||||
#ifdef CloseSocket
|
||||
#error "CloseSocket is a macro here (wolfssl/wolfio.h): it would bypass CloseSocket(SOCKET&) in util/sock.cpp"
|
||||
#endif
|
||||
using namespace hush;
|
||||
extern int32_t HUSH_TESTNODE;
|
||||
extern uint8_t ASSETCHAINS_CLEARNET;
|
||||
@@ -2958,6 +2951,16 @@ void CNode::PushAddrMessage(CSerializedNetMsg&& msg)
|
||||
}
|
||||
}
|
||||
|
||||
void GetNodeCounts(int& nAll, int& nTls)
|
||||
{
|
||||
LOCK(cs_vNodes);
|
||||
nAll = (int)vNodes.size();
|
||||
nTls = 0;
|
||||
for (const CNode* n : vNodes)
|
||||
if (n->ssl != NULL)
|
||||
nTls++;
|
||||
}
|
||||
|
||||
size_t GetNodeCount(NumConnections flags)
|
||||
{
|
||||
LOCK(cs_vNodes);
|
||||
|
||||
@@ -57,14 +57,6 @@
|
||||
#define byte unsigned char
|
||||
#include <wolfssl/ssl.h>
|
||||
#undef byte
|
||||
// wolfssl/wolfio.h (pulled in by ssl.h) does `#ifndef CloseSocket / #define CloseSocket(s) close(s)`. Left
|
||||
// defined, it turns every CloseSocket(hSocket) after this point -- all 15 calls in net.cpp, including the ones in
|
||||
// CloseSocketDisconnect and ~CNode -- into a bare close() that never sets hSocket = INVALID_SOCKET, so util/sock.cpp's
|
||||
// CloseSocket(SOCKET&) is never called. Every P2P disconnect then closed its fd number again later (a repeated
|
||||
// CloseSocketDisconnect, then ~CNode), by which time another thread often owned that number: wallet.dat or a
|
||||
// database/log.* fd (BDB then wrote wallet pages into whatever got the number next, e.g. a blocks/*.dat file
|
||||
// opened "rb+"), a block file, or an HTTP RPC socket. wolfio.h is include-guarded, so it can't come back.
|
||||
#undef CloseSocket
|
||||
|
||||
class CAddrMan;
|
||||
class CBlockIndex;
|
||||
@@ -173,6 +165,7 @@ enum NumConnections {
|
||||
};
|
||||
|
||||
size_t GetNodeCount(NumConnections num);
|
||||
void GetNodeCounts(int& nAll, int& nTls);
|
||||
|
||||
bool GetNetworkActive();
|
||||
void SetNetworkActive(bool active);
|
||||
|
||||
+6
-2
@@ -132,6 +132,10 @@ UniValue getinfo(const UniValue& params, bool fHelp, const CPubKey& mypk)
|
||||
+ HelpExampleRpc("getinfo", "")
|
||||
);
|
||||
|
||||
// C268: snapshot peer counts under cs_vNodes (sequentially, before the cs_main/cs_wallet lock)
|
||||
int nAllConns = 0, nTlsConns = 0;
|
||||
GetNodeCounts(nAllConns, nTlsConns);
|
||||
|
||||
#ifdef ENABLE_WALLET
|
||||
LOCK2(cs_main, pwalletMain ? &pwalletMain->cs_wallet : NULL);
|
||||
#else
|
||||
@@ -175,8 +179,8 @@ UniValue getinfo(const UniValue& params, bool fHelp, const CPubKey& mypk)
|
||||
}
|
||||
//obj.push_back(Pair("build_date", BUILD_DATE));
|
||||
obj.push_back(Pair("timeoffset", 0));
|
||||
obj.push_back(Pair("connections", (int)vNodes.size()));
|
||||
obj.push_back(Pair("tls_connections", (int)std::count_if(vNodes.begin(), vNodes.end(), [](CNode* n) {return n->ssl != NULL;})));
|
||||
obj.push_back(Pair("connections", nAllConns));
|
||||
obj.push_back(Pair("tls_connections", nTlsConns));
|
||||
obj.push_back(Pair("proxy", (proxy.IsValid() ? proxy.proxy.ToStringIPPort() : string())));
|
||||
obj.push_back(Pair("testnet", Params().TestnetToBeDeprecatedFieldRPC()));
|
||||
obj.push_back(Pair("relayfee", ValueFromAmount(::minRelayTxFee.GetFeePerK())));
|
||||
|
||||
+5
-2
@@ -533,6 +533,9 @@ UniValue getnetworkinfo(const UniValue& params, bool fHelp, const CPubKey& mypk)
|
||||
+ HelpExampleRpc("getnetworkinfo", "")
|
||||
);
|
||||
|
||||
int nAllConns = 0, nTlsConns = 0;
|
||||
GetNodeCounts(nAllConns, nTlsConns);
|
||||
|
||||
LOCK(cs_main);
|
||||
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
@@ -541,8 +544,8 @@ UniValue getnetworkinfo(const UniValue& params, bool fHelp, const CPubKey& mypk)
|
||||
obj.push_back(Pair("protocolversion",PROTOCOL_VERSION));
|
||||
obj.push_back(Pair("localservices", strprintf("%016x", nLocalServices)));
|
||||
obj.push_back(Pair("timeoffset", 0));
|
||||
obj.push_back(Pair("connections", (int)vNodes.size()));
|
||||
obj.push_back(Pair("tls_connections", (int)std::count_if(vNodes.begin(), vNodes.end(), [](CNode* n) {return n->ssl != NULL;})));
|
||||
obj.push_back(Pair("connections", nAllConns));
|
||||
obj.push_back(Pair("tls_connections", nTlsConns));
|
||||
obj.push_back(Pair("networks", GetNetworksInfo()));
|
||||
obj.push_back(Pair("relayfee", ValueFromAmount(::minRelayTxFee.GetFeePerK())));
|
||||
UniValue localAddresses(UniValue::VARR);
|
||||
|
||||
@@ -26,7 +26,6 @@
|
||||
#include "util.h"
|
||||
#include "util/strencodings.h"
|
||||
#include "asyncrpcqueue.h"
|
||||
#include <chrono>
|
||||
#include <memory>
|
||||
#include <univalue.h>
|
||||
#include <boost/bind.hpp>
|
||||
@@ -641,30 +640,6 @@ std::string JSONRPCExecBatch(const UniValue& vReq)
|
||||
return ret.write() + "\n";
|
||||
}
|
||||
|
||||
namespace {
|
||||
// Logs an RPC whose wall time reached 500ms, or every RPC under -debug=bench. That time includes
|
||||
// waiting for cs_main / cs_wallet behind block connection or another RPC, which the callee's own
|
||||
// timers can't see because they start once the locks are held. A long-poll getblocktemplate waits
|
||||
// for a new block with cs_main released, so it is only logged under -debug=bench. The method is
|
||||
// logged as method=<name>: a bare " <name> " token would match the GUI's " rescan " log heuristic.
|
||||
class RPCCallTimer {
|
||||
const std::string& strMethod;
|
||||
const bool fLongPoll;
|
||||
const std::chrono::steady_clock::time_point start;
|
||||
public:
|
||||
RPCCallTimer(const std::string& method, const UniValue& params)
|
||||
: strMethod(method),
|
||||
fLongPoll(method == "getblocktemplate" && params.size() > 0 && params[0].isObject() &&
|
||||
!find_value(params[0], "longpollid").isNull()),
|
||||
start(std::chrono::steady_clock::now()) {}
|
||||
~RPCCallTimer() {
|
||||
const int64_t nMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - start).count();
|
||||
if ((nMs >= 500 && !fLongPoll) || LogAcceptCategory("bench"))
|
||||
LogPrintf("RPC method=%s took %dms\n", strMethod, nMs);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
UniValue CRPCTable::execute(const std::string &strMethod, const UniValue ¶ms) const
|
||||
{
|
||||
// Find method
|
||||
@@ -699,7 +674,6 @@ UniValue CRPCTable::execute(const std::string &strMethod, const UniValue ¶ms
|
||||
try
|
||||
{
|
||||
// Execute
|
||||
RPCCallTimer timer(strMethod, params);
|
||||
return pcmd->actor(params, false, CPubKey());
|
||||
}
|
||||
catch (const std::exception& e)
|
||||
|
||||
+18
-53
@@ -86,30 +86,6 @@ struct CachedBlockData {
|
||||
pindex(pindex), oldTrees(oldTrees), txConflicted(txConflicted) {}
|
||||
};
|
||||
|
||||
bool NotifyWalletsOrShutdown(const std::function<void()>& notify)
|
||||
{
|
||||
// The wallet can throw here on a database failure, e.g. CWalletDB::ListAccountCreditDebit's "error scanning
|
||||
// DB" after BDB lost its file. Escaping this thread, that std::terminate()d the node (TraceThread logs and
|
||||
// rethrows) with nothing flushed. Stop cleanly instead: going on would leave the wallet behind the chain,
|
||||
// missing the block it failed on.
|
||||
try {
|
||||
notify();
|
||||
return true;
|
||||
} catch (const boost::thread_interrupted&) {
|
||||
throw;
|
||||
} catch (const std::exception& e) {
|
||||
PrintExceptionContinue(&e, "ThreadNotifyWallets()");
|
||||
} catch (...) {
|
||||
PrintExceptionContinue(NULL, "ThreadNotifyWallets()");
|
||||
}
|
||||
LogPrintf("*** %s\n", "Wallet failed to process a block notification; shutting down");
|
||||
StartShutdown();
|
||||
uiInterface.ThreadSafeMessageBox(
|
||||
_("Error: A fatal internal error occurred, see debug.log for details"),
|
||||
"", CClientUIInterface::MSG_ERROR);
|
||||
return false;
|
||||
}
|
||||
|
||||
void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
|
||||
{
|
||||
// If pindexLastTip == nullptr, the wallet is at genesis.
|
||||
@@ -227,21 +203,15 @@ void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
|
||||
_("Error: A fatal internal error occurred, see debug.log for details"),
|
||||
"", CClientUIInterface::MSG_ERROR);
|
||||
StartShutdown();
|
||||
// Don't hand the wallet an empty block as if it had been processed.
|
||||
fDisconnectIncomplete = true;
|
||||
break;
|
||||
}
|
||||
|
||||
if (!NotifyWalletsOrShutdown([&] {
|
||||
// Let wallets know transactions went from 1-confirmed to
|
||||
// 0-confirmed or conflicted:
|
||||
for (const CTransaction &tx : block.vtx) {
|
||||
SyncWithWallets(tx, NULL);
|
||||
}
|
||||
// Update cached incremental witnesses
|
||||
GetMainSignals().ChainTip(pindexLastTip, &block, boost::none);
|
||||
}))
|
||||
return;
|
||||
// Let wallets know transactions went from 1-confirmed to
|
||||
// 0-confirmed or conflicted:
|
||||
for (const CTransaction &tx : block.vtx) {
|
||||
SyncWithWallets(tx, NULL);
|
||||
}
|
||||
// Update cached incremental witnesses
|
||||
GetMainSignals().ChainTip(pindexLastTip, &block, boost::none);
|
||||
|
||||
// On to the next block!
|
||||
pindexLastTip = pindexLastTip->pprev;
|
||||
@@ -269,24 +239,19 @@ void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
|
||||
_("Error: A fatal internal error occurred, see debug.log for details"),
|
||||
"", CClientUIInterface::MSG_ERROR);
|
||||
StartShutdown();
|
||||
// Don't hand the wallet an empty block as if it had been processed.
|
||||
break;
|
||||
}
|
||||
|
||||
if (!NotifyWalletsOrShutdown([&] {
|
||||
// Tell wallet about transactions that went from mempool
|
||||
// to conflicted:
|
||||
for (const CTransaction &tx : blockData.txConflicted) {
|
||||
SyncWithWallets(tx, NULL);
|
||||
}
|
||||
// ... and about transactions that got confirmed:
|
||||
for (const CTransaction &tx : block.vtx) {
|
||||
SyncWithWallets(tx, &block);
|
||||
}
|
||||
// Update cached incremental witnesses
|
||||
GetMainSignals().ChainTip(blockData.pindex, &block, blockData.oldTrees);
|
||||
}))
|
||||
return;
|
||||
// Tell wallet about transactions that went from mempool
|
||||
// to conflicted:
|
||||
for (const CTransaction &tx : blockData.txConflicted) {
|
||||
SyncWithWallets(tx, NULL);
|
||||
}
|
||||
// ... and about transactions that got confirmed:
|
||||
for (const CTransaction &tx : block.vtx) {
|
||||
SyncWithWallets(tx, &block);
|
||||
}
|
||||
// Update cached incremental witnesses
|
||||
GetMainSignals().ChainTip(blockData.pindex, &block, blockData.oldTrees);
|
||||
|
||||
// This block is done!
|
||||
pindexLastTip = blockData.pindex;
|
||||
|
||||
@@ -8,7 +8,6 @@
|
||||
#define HUSH_VALIDATIONINTERFACE_H
|
||||
|
||||
#include <boost/signals2/signal.hpp>
|
||||
#include <functional>
|
||||
#include "zcash/IncrementalMerkleTree.hpp"
|
||||
|
||||
class CBlock;
|
||||
@@ -77,8 +76,4 @@ CMainSignals& GetMainSignals();
|
||||
|
||||
void ThreadNotifyWallets(CBlockIndex *pindexLastTip);
|
||||
|
||||
/** Runs one block's wallet notifications for ThreadNotifyWallets. Returns false, after logging the error and
|
||||
* requesting a shutdown, if they threw (e.g. a wallet database failure); thread interruption propagates. */
|
||||
bool NotifyWalletsOrShutdown(const std::function<void()>& notify);
|
||||
|
||||
#endif // HUSH_VALIDATIONINTERFACE_H
|
||||
|
||||
+1
-7
@@ -22,13 +22,7 @@
|
||||
|
||||
// network protocol versioning
|
||||
// DragonX 1.0.0 - bumped to separate from old HUSH/DragonX nodes with RandomX bug
|
||||
// DragonX 1.3.0 - bumped for the consensus changes in this release (BLOCK_VALID_CONTEXT
|
||||
// no longer overwrites the validity level, the dead CBOPRET coinbase
|
||||
// price check is gone, and pow.cpp lost 157 lines). MIN_PEER_PROTO_VERSION
|
||||
// is deliberately NOT raised: it currently equals this value, and raising
|
||||
// it would disconnect every node still on v1.0.3 -- which is all users,
|
||||
// since no release has ever been installable (unsigned archives).
|
||||
static const int PROTOCOL_VERSION = 2000001;
|
||||
static const int PROTOCOL_VERSION = 2000000;
|
||||
//! initial proto version, to be increased after version/verack negotiation
|
||||
static const int INIT_PROTO_VERSION = 209;
|
||||
//! In this version, 'getheaders' was introduced.
|
||||
|
||||
@@ -591,7 +591,7 @@ bool CCryptoKeyStore::GetSaplingSpendingKey(const libzcash::SaplingFullViewingKe
|
||||
if (!IsCrypted())
|
||||
return CBasicKeyStore::GetSaplingSpendingKey(fvk, skOut);
|
||||
|
||||
for (const auto& entry : mapCryptedSaplingSpendingKeys) {
|
||||
for (auto entry : mapCryptedSaplingSpendingKeys) {
|
||||
if (entry.first.fvk == fvk) {
|
||||
const std::vector<unsigned char> &vchCryptedSecret = entry.second;
|
||||
return DecryptSaplingSpendingKey(vMasterKey, vchCryptedSecret, entry.first, skOut);
|
||||
|
||||
@@ -265,7 +265,7 @@ public:
|
||||
LOCK(cs_SpendingKeyStore);
|
||||
if (!IsCrypted())
|
||||
return CBasicKeyStore::HaveSaplingSpendingKey(fvk);
|
||||
for (const auto& entry : mapCryptedSaplingSpendingKeys) {
|
||||
for (auto entry : mapCryptedSaplingSpendingKeys) {
|
||||
if (entry.first.fvk == fvk) {
|
||||
return true;
|
||||
}
|
||||
|
||||
+9
-32
@@ -61,7 +61,6 @@ void CDBEnv::Reset()
|
||||
dbenv = new DbEnv(DB_CXX_NO_EXCEPTIONS);
|
||||
fDbEnvInit = false;
|
||||
fMockDb = false;
|
||||
fKeepLogs = false;
|
||||
}
|
||||
|
||||
CDBEnv::CDBEnv() : dbenv(NULL)
|
||||
@@ -277,12 +276,10 @@ bool CDBEnv::Salvage(const std::string& strFile, bool fAggressive, std::vector<C
|
||||
|
||||
void CDBEnv::CheckpointLSN(const std::string& strFile)
|
||||
{
|
||||
if (dbenv->txn_checkpoint(0, 0, 0) != 0)
|
||||
fKeepLogs = true;
|
||||
dbenv->txn_checkpoint(0, 0, 0);
|
||||
if (fMockDb)
|
||||
return;
|
||||
if (dbenv->lsn_reset(strFile.c_str(), 0) != 0)
|
||||
fKeepLogs = true;
|
||||
dbenv->lsn_reset(strFile.c_str(), 0);
|
||||
}
|
||||
|
||||
|
||||
@@ -501,16 +498,10 @@ void CDBEnv::Flush(bool fShutdown)
|
||||
// Move log data to the dat file
|
||||
CloseDb(strFile);
|
||||
LogPrint("db", "CDBEnv::Flush: %s checkpoint\n", strFile);
|
||||
int ret = dbenv->txn_checkpoint(0, 0, 0);
|
||||
if (ret != 0) {
|
||||
LogPrintf("CDBEnv::Flush: %s checkpoint failed: %s\n", strFile, DbEnv::strerror(ret));
|
||||
fKeepLogs = true;
|
||||
}
|
||||
dbenv->txn_checkpoint(0, 0, 0);
|
||||
LogPrint("db", "CDBEnv::Flush: %s detach\n", strFile);
|
||||
if (!fMockDb && (ret = dbenv->lsn_reset(strFile.c_str(), 0)) != 0) {
|
||||
LogPrintf("CDBEnv::Flush: %s lsn_reset failed: %s\n", strFile, DbEnv::strerror(ret));
|
||||
fKeepLogs = true;
|
||||
}
|
||||
if (!fMockDb)
|
||||
dbenv->lsn_reset(strFile.c_str(), 0);
|
||||
LogPrint("db", "CDBEnv::Flush: %s closed\n", strFile);
|
||||
mapFileUseCount.erase(mi++);
|
||||
} else
|
||||
@@ -520,24 +511,10 @@ void CDBEnv::Flush(bool fShutdown)
|
||||
if (fShutdown) {
|
||||
char** listp;
|
||||
if (mapFileUseCount.empty()) {
|
||||
// After a failed flush, or once the environment has panicked (BDB0060, e.g. after a page
|
||||
// write failed), wallet.dat can be missing committed changes that only the logs hold.
|
||||
// Removing them would turn a wallet the next start recovers (DB_RECOVER) into one that
|
||||
// silently lost them. If the logs are unusable, the next start moves database/ aside.
|
||||
// (get_flags itself fails with DB_RUNRECOVERY once the environment has panicked.)
|
||||
u_int32_t envFlags = 0;
|
||||
if (dbenv->get_flags(&envFlags) != 0 || (envFlags & DB_PANIC_ENVIRONMENT))
|
||||
fKeepLogs = true;
|
||||
if (fKeepLogs && !fMockDb) {
|
||||
LogPrintf("CDBEnv::Flush: the wallet database was not flushed cleanly; keeping %s for recovery at the next start\n",
|
||||
(boost::filesystem::path(strPath) / "database").string());
|
||||
Close();
|
||||
} else {
|
||||
dbenv->log_archive(&listp, DB_ARCH_REMOVE);
|
||||
Close();
|
||||
if (!fMockDb)
|
||||
boost::filesystem::remove_all(boost::filesystem::path(strPath) / "database");
|
||||
}
|
||||
dbenv->log_archive(&listp, DB_ARCH_REMOVE);
|
||||
Close();
|
||||
if (!fMockDb)
|
||||
boost::filesystem::remove_all(boost::filesystem::path(strPath) / "database");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -28,7 +28,6 @@
|
||||
#include "sync.h"
|
||||
#include "version.h"
|
||||
|
||||
#include <atomic>
|
||||
#include <map>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
@@ -54,9 +53,6 @@ class CDBEnv
|
||||
private:
|
||||
bool fDbEnvInit;
|
||||
bool fMockDb;
|
||||
// A checkpoint or lsn_reset failed: the logs in database/ may hold the only copy of committed
|
||||
// wallet changes, so the shutdown flush must leave them for DB_RECOVER at the next start.
|
||||
std::atomic<bool> fKeepLogs;
|
||||
// Don't change into boost::filesystem::path, as that can result in
|
||||
// shutdown problems/crashes caused by a static initialized internal pointer.
|
||||
std::string strPath;
|
||||
|
||||
+139
-155
@@ -11,112 +11,70 @@
|
||||
#include "utilmoneystr.h"
|
||||
#include <utf8.h>
|
||||
|
||||
#include <map>
|
||||
#include <set>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
using namespace std;
|
||||
using namespace libzcash;
|
||||
|
||||
bool EnsureWalletIsAvailable(bool avoidException);
|
||||
|
||||
SaplingHistoryKeys GetSaplingHistoryKeys()
|
||||
const std::vector<ZsTxJSONContext::Key>& ZsTxJSONContext::keys()
|
||||
{
|
||||
AssertLockHeld(pwalletMain->cs_wallet);
|
||||
SaplingHistoryKeys keys;
|
||||
std::set<libzcash::SaplingPaymentAddress> addresses;
|
||||
pwalletMain->GetSaplingPaymentAddresses(addresses);
|
||||
for (const auto& addr : addresses) {
|
||||
libzcash::SaplingExtendedSpendingKey extsk;
|
||||
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
|
||||
const libzcash::SaplingFullViewingKey fvk = extsk.expsk.full_viewing_key();
|
||||
SaplingHistoryKeys::SpendableAddress a;
|
||||
a.addr = addr;
|
||||
a.ivk = fvk.in_viewing_key();
|
||||
a.ovk = fvk.ovk;
|
||||
// Left zero when the wallet doesn't know addr, exactly as GetNullifiersForAddresses does.
|
||||
pwalletMain->GetSaplingIncomingViewingKey(addr, a.walletIvk);
|
||||
keys.spendable.push_back(a);
|
||||
if (!keysBuilt) {
|
||||
// The calls, order and filter of the per-output key loops this replaces.
|
||||
std::set<libzcash::SaplingPaymentAddress> addresses;
|
||||
pwalletMain->GetSaplingPaymentAddresses(addresses);
|
||||
for (std::set<libzcash::SaplingPaymentAddress>::const_iterator it = addresses.begin(); it != addresses.end(); ++it) {
|
||||
libzcash::SaplingExtendedSpendingKey extsk;
|
||||
if (!pwalletMain->GetSaplingExtendedSpendingKey(*it, extsk))
|
||||
continue; // viewing-key-only, or encrypted and locked: skipped, as before
|
||||
libzcash::SaplingFullViewingKey fvk = extsk.expsk.full_viewing_key();
|
||||
Key key;
|
||||
key.addr = *it;
|
||||
key.ivk = fvk.in_viewing_key();
|
||||
key.ovk = fvk.ovk;
|
||||
vKeys.push_back(key);
|
||||
}
|
||||
keysBuilt = true;
|
||||
}
|
||||
}
|
||||
return keys;
|
||||
return vKeys;
|
||||
}
|
||||
|
||||
bool SaplingHistoryKeys::IsNullifierOfAddress(const libzcash::SaplingIncomingViewingKey& addrIvk, const uint256& nullifier) const
|
||||
bool ZsTxJSONContext::hasNullifier(const libzcash::SaplingPaymentAddress& addr, const uint256& nullifier)
|
||||
{
|
||||
AssertLockHeld(pwalletMain->cs_wallet);
|
||||
if (!noteNullifiers) {
|
||||
noteNullifiers = std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256>>();
|
||||
for (const auto& p : pwalletMain->mapWallet) {
|
||||
for (const auto& item : p.second.mapSaplingNoteData) {
|
||||
if (item.second.nullifier)
|
||||
noteNullifiers->insert(std::make_pair(item.second.ivk, *item.second.nullifier));
|
||||
}
|
||||
if (!nullifiersBuilt) {
|
||||
// The population GetNullifiersForAddresses() scans: every mapWallet note that has a nullifier.
|
||||
for (std::map<uint256, CWalletTx>::const_iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
|
||||
for (mapSaplingNoteData_t::const_iterator nit = it->second.mapSaplingNoteData.begin(); nit != it->second.mapSaplingNoteData.end(); ++nit) {
|
||||
if (nit->second.nullifier)
|
||||
nfByIvk.insert(std::make_pair(nit->second.ivk, nit->second.nullifier.get()));
|
||||
}
|
||||
}
|
||||
nullifiersBuilt = true;
|
||||
}
|
||||
}
|
||||
return noteNullifiers->count(std::make_pair(addrIvk, nullifier)) > 0;
|
||||
// GetNullifiersForAddresses() ignores the result of this lookup, so an address the wallet does not
|
||||
// know is matched against the default (null) ivk. Keep that.
|
||||
libzcash::SaplingIncomingViewingKey ivk;
|
||||
pwalletMain->GetSaplingIncomingViewingKey(addr, ivk);
|
||||
return nfByIvk.count(std::make_pair(ivk, nullifier)) > 0;
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
// One output's decryptions under incoming viewing keys, each key tried at most once. Two distinct ivks can't
|
||||
// both decrypt an output: the cm check recomputes pk_d = [ivk]g_d, and ivks are canonical scalars below the
|
||||
// order of g_d (ka_agree and ivk_to_pkd reject the rest), so a second one would need a NoteCommit collision.
|
||||
// So once it is known which ivk decrypts the output (the wallet's note data records the one FindMySaplingNotes
|
||||
// found, with its plaintext normally cached), no other ivk is tried.
|
||||
class OutputDecryptions {
|
||||
const OutputDescription& output;
|
||||
const SaplingNoteData* nd;
|
||||
std::map<libzcash::SaplingIncomingViewingKey, boost::optional<SaplingNoteCache>> tried;
|
||||
|
||||
boost::optional<SaplingNoteCache> Decrypt(const libzcash::SaplingIncomingViewingKey& ivk) const {
|
||||
auto both = libzcash::SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
|
||||
if (!both)
|
||||
return boost::none;
|
||||
return SaplingNoteCache{ both->second, both->first.memo() };
|
||||
}
|
||||
// The plaintext under the ivk the wallet recorded for this output, decrypting it if it isn't cached.
|
||||
const boost::optional<SaplingNoteCache>& Recorded() {
|
||||
auto it = tried.find(nd->ivk);
|
||||
if (it == tried.end())
|
||||
it = tried.insert(std::make_pair(nd->ivk, nd->cachedPlaintext ? nd->cachedPlaintext : Decrypt(nd->ivk))).first;
|
||||
return it->second;
|
||||
}
|
||||
public:
|
||||
OutputDecryptions(const CWalletTx& wtx, uint32_t n) : output(wtx.vShieldedOutput[n]), nd(nullptr) {
|
||||
auto it = wtx.mapSaplingNoteData.find(SaplingOutPoint(wtx.GetHash(), n));
|
||||
if (it != wtx.mapSaplingNoteData.end())
|
||||
nd = &it->second;
|
||||
}
|
||||
const SaplingNoteCache* Get(const libzcash::SaplingIncomingViewingKey& ivk) {
|
||||
auto it = tried.find(ivk);
|
||||
if (it == tried.end()) {
|
||||
boost::optional<SaplingNoteCache> result;
|
||||
if (nd && nd->ivk == ivk) {
|
||||
result = Recorded();
|
||||
} else if (!(nd && Recorded())) {
|
||||
// Not a wallet note, or one whose recorded ivk doesn't decrypt it: try this key.
|
||||
result = Decrypt(ivk);
|
||||
}
|
||||
it = tried.insert(std::make_pair(ivk, result)).first;
|
||||
}
|
||||
return it->second ? &*it->second : nullptr;
|
||||
}
|
||||
};
|
||||
|
||||
// What an outCiphertext tells the sender holding ovk: the note's plaintext and the address it was sent to.
|
||||
boost::optional<std::pair<libzcash::SaplingNotePlaintext, libzcash::SaplingPaymentAddress>> DecryptSent(
|
||||
const OutputDescription& outputDesc, const uint256& ovk)
|
||||
bool ZsTxJSONContext::seedOvk(uint256& ovkOut)
|
||||
{
|
||||
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
|
||||
outputDesc.outCiphertext,ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
|
||||
if (!opt)
|
||||
return boost::none;
|
||||
auto opt_unwrapped = opt.get();
|
||||
auto pt = libzcash::SaplingNotePlaintext::decrypt(
|
||||
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
|
||||
if (!pt)
|
||||
return boost::none;
|
||||
auto pt_unwrapped = pt.get();
|
||||
return std::make_pair(pt_unwrapped, libzcash::SaplingPaymentAddress(pt_unwrapped.d, opt_unwrapped.pk_d));
|
||||
}
|
||||
|
||||
if (!seedOvkTried) {
|
||||
HDSeed seed;
|
||||
if (pwalletMain->GetHDSeedForDerivation(seed)) {
|
||||
seedOvkValue = ovkForShieldingFromTaddr(seed);
|
||||
seedOvkOk = true;
|
||||
}
|
||||
seedOvkTried = true;
|
||||
}
|
||||
if (seedOvkOk)
|
||||
ovkOut = seedOvkValue;
|
||||
return seedOvkOk;
|
||||
}
|
||||
|
||||
void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) {
|
||||
@@ -190,29 +148,22 @@ void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpen
|
||||
const SpendDescription& spendDesc = wtx.vShieldedSpend[i];
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
|
||||
// Lookups by find(), not operator[]: that inserted an entry for every other wallet's nullifier it met
|
||||
// (and could insert an empty note into a wallet tx).
|
||||
if (pwalletMain->IsSaplingNullifierFromMe(spendDesc.nullifier)) {
|
||||
const SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes.at(spendDesc.nullifier);
|
||||
std::map<uint256, SaplingOutPoint>::const_iterator nfIt = pwalletMain->mapSaplingNullifiersToNotes.find(spendDesc.nullifier);
|
||||
|
||||
// Same test as IsSaplingNullifierFromMe(), but find() so a read-only RPC never inserts into wallet maps.
|
||||
if (nfIt != pwalletMain->mapSaplingNullifiersToNotes.end() && pwalletMain->mapWallet.count(nfIt->second.hash)) {
|
||||
const SaplingOutPoint op = nfIt->second;
|
||||
const CWalletTx* parent = pwalletMain->GetWalletTx(op.hash);
|
||||
auto ndIt = parent->mapSaplingNoteData.find(op);
|
||||
// The spent note's plaintext, from the wallet's cache when it has it.
|
||||
boost::optional<SaplingNoteCache> pt;
|
||||
if (ndIt != parent->mapSaplingNoteData.end() && op.n < parent->vShieldedOutput.size()) {
|
||||
const SaplingNoteData& nd = ndIt->second;
|
||||
if (nd.cachedPlaintext) {
|
||||
pt = nd.cachedPlaintext;
|
||||
} else {
|
||||
const OutputDescription& output = parent->vShieldedOutput[op.n];
|
||||
auto both = libzcash::SaplingNotePlaintext::decrypt_with_note(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
|
||||
if (both)
|
||||
pt = SaplingNoteCache{ both->second, both->first.memo() };
|
||||
}
|
||||
}
|
||||
mapSaplingNoteData_t::const_iterator ndIt = parent->mapSaplingNoteData.find(op);
|
||||
if (ndIt == parent->mapSaplingNoteData.end()) continue; // stale entry: old code inserted an empty note that never decrypts
|
||||
const OutputDescription& output = parent->vShieldedOutput[op.n];
|
||||
const SaplingNoteData& nd = ndIt->second;
|
||||
auto pt = libzcash::SaplingNotePlaintext::decrypt(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
|
||||
|
||||
if (pt) {
|
||||
const libzcash::SaplingNote& note = pt->note;
|
||||
const libzcash::SaplingPaymentAddress address(note.d, note.pk_d);
|
||||
auto note = pt.get();
|
||||
auto pa = nd.ivk.address(note.d);
|
||||
auto address = pa.get();
|
||||
shieldedSpendsSpentCount++;
|
||||
totalSpends += CAmount(-note.value());
|
||||
obj.push_back(Pair("address",EncodePaymentAddress(address)));
|
||||
@@ -246,13 +197,16 @@ void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpen
|
||||
}
|
||||
|
||||
|
||||
void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys) {
|
||||
void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, ZsTxJSONContext* ctx) {
|
||||
|
||||
if(fZdebug)
|
||||
fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
|
||||
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
|
||||
ZsTxJSONContext localCtx; // used only when the caller passed no per-call memo
|
||||
ZsTxJSONContext& zctx = ctx ? *ctx : localCtx;
|
||||
|
||||
//Check address
|
||||
bool isTAddress = false;
|
||||
bool isZsAddress = false;
|
||||
@@ -302,18 +256,45 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
|
||||
UniValue zsReceived(UniValue::VARR);
|
||||
if (isZsAddress || !filterByAddress) {
|
||||
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
|
||||
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
bool changeTx = false;
|
||||
//Decrypt sapling incoming commitments using IVK: once per key, not once per address
|
||||
OutputDecryptions decryptions(wtx, i);
|
||||
for (const SaplingHistoryKeys::SpendableAddress& a : keys.spendable) {
|
||||
const libzcash::SaplingPaymentAddress& addr = a.addr;
|
||||
{
|
||||
const SaplingNoteCache* pt = decryptions.Get(a.ivk);
|
||||
//Decrypt sapling incoming commitments using IVK
|
||||
const std::vector<ZsTxJSONContext::Key>& keys = zctx.keys();
|
||||
// An output opens under at most one ivk (decrypt() checks the note commitment, which binds
|
||||
// pk_d = [ivk]g_d), and FindMySaplingNotes records that ivk in mapSaplingNoteData. When the recorded
|
||||
// ivk is one of keys(), pass 0 tries only the keys with that ivk: every other key would fail, as in
|
||||
// the old single loop. If pass 0 opens nothing (stale note data), pass 1 tries the remaining keys in
|
||||
// the same order. Without a usable record only pass 1 runs, over every key, exactly as before.
|
||||
libzcash::SaplingIncomingViewingKey recordedIvk;
|
||||
bool narrow = false;
|
||||
{
|
||||
mapSaplingNoteData_t::const_iterator ndIt = wtx.mapSaplingNoteData.find(SaplingOutPoint(wtx.GetHash(), i));
|
||||
if (ndIt != wtx.mapSaplingNoteData.end()) {
|
||||
for (size_t k = 0; k < keys.size() && !narrow; k++) {
|
||||
if (keys[k].ivk == ndIt->second.ivk) {
|
||||
recordedIvk = ndIt->second.ivk;
|
||||
narrow = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
bool openedByRecordedIvk = false;
|
||||
for (int pass = narrow ? 0 : 1; pass < 2 && !openedByRecordedIvk; pass++) {
|
||||
for (size_t k = 0; k < keys.size(); k++) {
|
||||
if (narrow && ((pass == 0) != (keys[k].ivk == recordedIvk)))
|
||||
continue; // pass 0: only the recorded ivk; pass 1: only the others
|
||||
const libzcash::SaplingPaymentAddress& addr = keys[k].addr;
|
||||
{ // spending-key availability is resolved once per call, in ZsTxJSONContext::keys()
|
||||
auto pt = libzcash::SaplingNotePlaintext::decrypt(
|
||||
outputDesc.encCiphertext, keys[k].ivk, outputDesc.ephemeralKey, outputDesc.cm);
|
||||
if (pt && pass == 0)
|
||||
openedByRecordedIvk = true;
|
||||
|
||||
if (pt) {
|
||||
const libzcash::SaplingNote& note = pt->note;
|
||||
auto memo = pt->memo;
|
||||
auto note = pt.get();
|
||||
auto pt_unwrapped = pt.get();
|
||||
auto memo = pt_unwrapped.memo();
|
||||
obj.push_back(Pair("address",EncodePaymentAddress(addr)));
|
||||
obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value()))));
|
||||
obj.push_back(Pair("memo", HexStr(memo)));
|
||||
@@ -330,7 +311,7 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
|
||||
//Check Change Status
|
||||
if (wtx.vShieldedSpend.size()!=0) {
|
||||
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
|
||||
if (keys.IsNullifierOfAddress(a.walletIvk, spendDesc.nullifier)) {
|
||||
if (zctx.hasNullifier(addr, spendDesc.nullifier)) {
|
||||
changeTx = true;
|
||||
}
|
||||
}
|
||||
@@ -343,6 +324,7 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
|
||||
}
|
||||
}
|
||||
}
|
||||
} // pass
|
||||
}
|
||||
}
|
||||
received.push_back(Pair("saplingReceived",zsReceived));
|
||||
@@ -351,10 +333,13 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
|
||||
}
|
||||
|
||||
|
||||
void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys) {
|
||||
void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, ZsTxJSONContext* ctx) {
|
||||
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
|
||||
ZsTxJSONContext localCtx; // used only when the caller passed no per-call memo
|
||||
ZsTxJSONContext& zctx = ctx ? *ctx : localCtx;
|
||||
|
||||
//Used to identify incomplete key sets
|
||||
int shieldedOutputCount = 0;
|
||||
int shieldedOutputDecryptedCount = 0;
|
||||
@@ -404,13 +389,6 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
|
||||
//All Shielded Sends in the transaction
|
||||
UniValue zsSends(UniValue::VARR);
|
||||
if (isZsAddress || !filterByAddress) {
|
||||
//The t to z shielding ovk comes from the HD seed: derive it once per tx, not once per output
|
||||
boost::optional<uint256> seedOvk;
|
||||
if (wtx.vShieldedSpend.size()==0 && wtx.vShieldedOutput.size()!=0) {
|
||||
HDSeed seed;
|
||||
if (pwalletMain->GetHDSeedForDerivation(seed))
|
||||
seedOvk = ovkForShieldingFromTaddr(seed);
|
||||
}
|
||||
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
|
||||
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
|
||||
shieldedOutputCount++;
|
||||
@@ -419,14 +397,20 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
|
||||
|
||||
//Decrypt sapling outgoing t to z transaction using HDseed
|
||||
if (wtx.vShieldedSpend.size()==0) {
|
||||
if (seedOvk) {
|
||||
{
|
||||
auto sent = DecryptSent(outputDesc, *seedOvk);
|
||||
uint256 taddrOvk;
|
||||
if (zctx.seedOvk(taddrOvk)) {
|
||||
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
|
||||
outputDesc.outCiphertext,taddrOvk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
|
||||
|
||||
if (sent) {
|
||||
if (opt) {
|
||||
auto opt_unwrapped = opt.get();
|
||||
auto pt = libzcash::SaplingNotePlaintext::decrypt(
|
||||
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
|
||||
|
||||
if (pt) {
|
||||
shieldedOutputDecryptedCount++;
|
||||
const libzcash::SaplingNotePlaintext& pt_unwrapped = sent->first;
|
||||
const libzcash::SaplingPaymentAddress& sentAddr = sent->second;
|
||||
auto pt_unwrapped = pt.get();
|
||||
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
|
||||
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
|
||||
obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value()))));
|
||||
obj.push_back(Pair("shieldedOutputIndex",i));
|
||||
@@ -439,22 +423,24 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
|
||||
}
|
||||
}
|
||||
|
||||
//attempt Decryption of Outgoing Sapling using wallet extended spending keys: once per ovk, not once per address
|
||||
//attempt Decryption of Outgoing Sapling using wallet extended spending keys
|
||||
} else {
|
||||
std::map<uint256, boost::optional<std::pair<libzcash::SaplingNotePlaintext, libzcash::SaplingPaymentAddress>>> sentByOvk;
|
||||
for (const SaplingHistoryKeys::SpendableAddress& a : keys.spendable) {
|
||||
{
|
||||
auto tried = sentByOvk.find(a.ovk);
|
||||
if (tried == sentByOvk.end())
|
||||
tried = sentByOvk.insert(std::make_pair(a.ovk, DecryptSent(outputDesc, a.ovk))).first;
|
||||
const auto& sent = tried->second;
|
||||
const std::vector<ZsTxJSONContext::Key>& keys = zctx.keys();
|
||||
for (size_t k = 0; k < keys.size(); k++) {
|
||||
{ // spending-key availability is resolved once per call, in ZsTxJSONContext::keys()
|
||||
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
|
||||
outputDesc.outCiphertext,keys[k].ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
|
||||
|
||||
{
|
||||
if (sent) {
|
||||
const libzcash::SaplingNotePlaintext& pt_unwrapped = sent->first;
|
||||
if (opt) {
|
||||
auto opt_unwrapped = opt.get();
|
||||
auto pt = libzcash::SaplingNotePlaintext::decrypt(
|
||||
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
|
||||
|
||||
if (pt) {
|
||||
auto pt_unwrapped = pt.get();
|
||||
auto memo = pt_unwrapped.memo();
|
||||
shieldedOutputDecryptedCount++;
|
||||
const libzcash::SaplingPaymentAddress& sentAddr = sent->second;
|
||||
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
|
||||
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
|
||||
obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value()))));
|
||||
obj.push_back(Pair("memo", HexStr(memo)));
|
||||
@@ -470,10 +456,8 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
|
||||
|
||||
//Check Change Status
|
||||
if (wtx.vShieldedSpend.size()!=0) {
|
||||
libzcash::SaplingIncomingViewingKey sentIvk;
|
||||
pwalletMain->GetSaplingIncomingViewingKey(sentAddr, sentIvk);
|
||||
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
|
||||
if (keys.IsNullifierOfAddress(sentIvk, spendDesc.nullifier)) {
|
||||
if (zctx.hasNullifier(sentAddr, spendDesc.nullifier)) {
|
||||
changeTx = true;
|
||||
}
|
||||
}
|
||||
@@ -502,7 +486,7 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
|
||||
}
|
||||
|
||||
|
||||
void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, const SaplingHistoryKeys& keys) {
|
||||
void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, ZsTxJSONContext* ctx) {
|
||||
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
|
||||
@@ -550,7 +534,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
|
||||
tx.push_back(Pair("expiryheight", (int64_t)wtx.nExpiryHeight));
|
||||
tx.push_back(Pair("confirmations", confirms));
|
||||
tx.push_back(Pair("time", wtx.GetTxTime()));
|
||||
tx.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
|
||||
tx.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<const CTransaction&>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
|
||||
|
||||
//Wallet Conflicts
|
||||
UniValue conflicts(UniValue::VARR);
|
||||
@@ -574,7 +558,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
|
||||
}
|
||||
// Get Received
|
||||
if (returnType == 0 || returnType == 2) {
|
||||
zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool, keys);
|
||||
zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool, ctx);
|
||||
if (!fBool || totalReceived != 0) {
|
||||
tx.push_back(Pair("received",received));
|
||||
}
|
||||
@@ -584,7 +568,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
|
||||
if (returnType == 0 || returnType == 3) {
|
||||
//Only include sends if there are spends that belong to the wallet.
|
||||
if (totalSpends != 0 || fBool) {
|
||||
zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool, keys);
|
||||
zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool, ctx);
|
||||
}
|
||||
if (!fBool || totalSends != 0) {
|
||||
tx.push_back(Pair("sends",sends));
|
||||
|
||||
+41
-23
@@ -6,6 +6,13 @@
|
||||
#ifndef HUSH_WALLET_RPCHUSHWALLET_H
|
||||
#define HUSH_WALLET_RPCHUSHWALLET_H
|
||||
|
||||
#include "uint256.h"
|
||||
#include "zcash/Address.hpp"
|
||||
|
||||
#include <set>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
struct balancestruct {
|
||||
CAmount confirmed;
|
||||
CAmount unconfirmed;
|
||||
@@ -13,35 +20,46 @@ struct balancestruct {
|
||||
CAmount immature;
|
||||
};
|
||||
|
||||
/** The wallet's Sapling keys as the history helpers use them, gathered once per RPC call. The helpers used to
|
||||
* look them up for every output and every address, re-deriving each full viewing key (an elliptic-curve
|
||||
* operation), and walked every note in the wallet for each change check. Build it with pwalletMain->cs_wallet
|
||||
* held, and keep holding it while the helpers run: that keeps the wallet's keys and notes fixed. The one thing
|
||||
* it can't stop is the walletpassphrase timer relocking the wallet mid-call; the call then reports the keys as
|
||||
* they were when it started, where the old per-output lookups switched over partway. */
|
||||
struct SaplingHistoryKeys {
|
||||
struct SpendableAddress {
|
||||
// Per-RPC-call memo for the zs*ToJSON helpers: the wallet's usable Sapling keys, the HD-seed ovk and a
|
||||
// nullifier index, derived once per call instead of once per output. Every member is a copy; nothing
|
||||
// points into wallet maps. Valid only while the caller holds LOCK2(cs_main, pwalletMain->cs_wallet) for
|
||||
// the whole call: declare it after that lock and never keep it across calls. Passing NULL to the helpers
|
||||
// gives each helper call its own memo (the old per-call cost profile, minus the per-output repeats).
|
||||
class ZsTxJSONContext {
|
||||
public:
|
||||
struct Key {
|
||||
libzcash::SaplingPaymentAddress addr;
|
||||
libzcash::SaplingIncomingViewingKey ivk; //!< from its spending key: what received outputs are decrypted with
|
||||
uint256 ovk; //!< from its spending key: what outCiphertexts are decrypted with
|
||||
libzcash::SaplingIncomingViewingKey walletIvk; //!< GetSaplingIncomingViewingKey(addr), as the change check looked it up
|
||||
libzcash::SaplingIncomingViewingKey ivk;
|
||||
uint256 ovk;
|
||||
};
|
||||
//! Every wallet address whose spending key is available (never while the wallet is locked), in
|
||||
//! GetSaplingPaymentAddresses order.
|
||||
std::vector<SpendableAddress> spendable;
|
||||
//! Whether GetNullifiersForAddresses({addr}) would contain (addr, nullifier), given addr's
|
||||
//! GetSaplingIncomingViewingKey result (zero when the wallet doesn't know addr): nullifier belongs to a
|
||||
//! wallet note received with that ivk. Walks the wallet's notes once, on first use.
|
||||
bool IsNullifierOfAddress(const libzcash::SaplingIncomingViewingKey& addrIvk, const uint256& nullifier) const;
|
||||
|
||||
ZsTxJSONContext() : keysBuilt(false), nullifiersBuilt(false), seedOvkTried(false), seedOvkOk(false) {}
|
||||
ZsTxJSONContext(const ZsTxJSONContext&) = delete;
|
||||
ZsTxJSONContext& operator=(const ZsTxJSONContext&) = delete;
|
||||
|
||||
// Wallet z-addresses (std::set order) whose extended spending key is available now, with that key's
|
||||
// extsk.expsk.full_viewing_key().in_viewing_key() and .ovk. Empty for an encrypted+locked wallet or
|
||||
// one that holds only viewing keys.
|
||||
const std::vector<Key>& keys();
|
||||
// == GetNullifiersForAddresses({addr}).count(std::make_pair(addr, nullifier)) > 0
|
||||
bool hasNullifier(const libzcash::SaplingPaymentAddress& addr, const uint256& nullifier);
|
||||
// == GetHDSeedForDerivation(seed) ? (ovkOut = ovkForShieldingFromTaddr(seed), true) : false
|
||||
bool seedOvk(uint256& ovkOut);
|
||||
|
||||
private:
|
||||
mutable boost::optional<std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256>>> noteNullifiers;
|
||||
bool keysBuilt;
|
||||
std::vector<Key> vKeys;
|
||||
bool nullifiersBuilt;
|
||||
std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256> > nfByIvk;
|
||||
bool seedOvkTried;
|
||||
bool seedOvkOk;
|
||||
uint256 seedOvkValue;
|
||||
};
|
||||
SaplingHistoryKeys GetSaplingHistoryKeys();
|
||||
|
||||
void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress);
|
||||
void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys);
|
||||
void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys);
|
||||
void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, const SaplingHistoryKeys& keys);
|
||||
void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, ZsTxJSONContext* ctx = NULL);
|
||||
void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, ZsTxJSONContext* ctx = NULL);
|
||||
void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, ZsTxJSONContext* ctx = NULL);
|
||||
|
||||
|
||||
#endif //HUSH_WALLET_RPCWALLET_H
|
||||
|
||||
+179
-104
@@ -48,6 +48,7 @@
|
||||
#include <boost/assign/list_of.hpp>
|
||||
#include <utf8.h>
|
||||
#include <univalue.h>
|
||||
#include <algorithm>
|
||||
#include <numeric>
|
||||
#include "hush_defs.h"
|
||||
#include <string.h>
|
||||
@@ -1773,7 +1774,19 @@ static void MaybePushAddress(UniValue & entry, const CTxDestination &dest)
|
||||
}
|
||||
}
|
||||
|
||||
void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDepth, bool fLong, UniValue& ret, const isminefilter& filter)
|
||||
// listtransactions skips its first `from` entries without building them: while *pnSkip > 0, an
|
||||
// entry that would be emitted only decrements *pnSkip. The counter is shared across calls, so the
|
||||
// skipped entries are exactly the first `from` ones in emission order. NULL: emit everything.
|
||||
static bool ConsumeSkip(int* pnSkip)
|
||||
{
|
||||
if (pnSkip != NULL && *pnSkip > 0) {
|
||||
--*pnSkip;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDepth, bool fLong, UniValue& ret, const isminefilter& filter, int* pnSkip = NULL)
|
||||
{
|
||||
CAmount nFee;
|
||||
string strSentAccount;
|
||||
@@ -1787,11 +1800,18 @@ void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDe
|
||||
bool fAllAccounts = (strAccount == string("*"));
|
||||
bool involvesWatchonly = wtx.IsFromMe(ISMINE_WATCH_ONLY);
|
||||
|
||||
// "size" is a property of the transaction, not of an entry: compute it once, when the first
|
||||
// entry is built, through a const reference instead of a by-value CTransaction copy.
|
||||
bool fHaveTxSize = false;
|
||||
uint64_t nTxSize = 0;
|
||||
|
||||
// Sent
|
||||
if ((!listSent.empty() || nFee != 0) && (fAllAccounts || strAccount == strSentAccount))
|
||||
{
|
||||
BOOST_FOREACH(const COutputEntry& s, listSent)
|
||||
{
|
||||
if (ConsumeSkip(pnSkip))
|
||||
continue;
|
||||
UniValue entry(UniValue::VOBJ);
|
||||
if(involvesWatchonly || (::IsMine(*pwalletMain, s.destination) & ISMINE_WATCH_ONLY))
|
||||
entry.push_back(Pair("involvesWatchonly", true));
|
||||
@@ -1803,7 +1823,11 @@ void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDe
|
||||
entry.push_back(Pair("fee", ValueFromAmount(-nFee)));
|
||||
if (fLong)
|
||||
WalletTxToJSON(wtx, entry);
|
||||
entry.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
|
||||
if (!fHaveTxSize) {
|
||||
nTxSize = static_cast<uint64_t>(GetSerializeSize(static_cast<const CTransaction&>(wtx), SER_NETWORK, PROTOCOL_VERSION));
|
||||
fHaveTxSize = true;
|
||||
}
|
||||
entry.push_back(Pair("size", nTxSize));
|
||||
ret.push_back(entry);
|
||||
}
|
||||
}
|
||||
@@ -1818,6 +1842,8 @@ void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDe
|
||||
account = pwalletMain->mapAddressBook[r.destination].name;
|
||||
if (fAllAccounts || (account == strAccount))
|
||||
{
|
||||
if (ConsumeSkip(pnSkip))
|
||||
continue;
|
||||
UniValue entry(UniValue::VOBJ);
|
||||
if(involvesWatchonly || (::IsMine(*pwalletMain, r.destination) & ISMINE_WATCH_ONLY))
|
||||
entry.push_back(Pair("involvesWatchonly", true));
|
||||
@@ -1848,19 +1874,25 @@ void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDe
|
||||
entry.push_back(Pair("vout", r.vout));
|
||||
if (fLong)
|
||||
WalletTxToJSON(wtx, entry);
|
||||
entry.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
|
||||
if (!fHaveTxSize) {
|
||||
nTxSize = static_cast<uint64_t>(GetSerializeSize(static_cast<const CTransaction&>(wtx), SER_NETWORK, PROTOCOL_VERSION));
|
||||
fHaveTxSize = true;
|
||||
}
|
||||
entry.push_back(Pair("size", nTxSize));
|
||||
ret.push_back(entry);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void AcentryToJSON(const CAccountingEntry& acentry, const string& strAccount, UniValue& ret)
|
||||
void AcentryToJSON(const CAccountingEntry& acentry, const string& strAccount, UniValue& ret, int* pnSkip = NULL)
|
||||
{
|
||||
bool fAllAccounts = (strAccount == string("*"));
|
||||
|
||||
if (fAllAccounts || acentry.strAccount == strAccount)
|
||||
{
|
||||
if (ConsumeSkip(pnSkip))
|
||||
return;
|
||||
UniValue entry(UniValue::VOBJ);
|
||||
entry.push_back(Pair("account", acentry.strAccount));
|
||||
entry.push_back(Pair("category", "move"));
|
||||
@@ -1954,46 +1986,42 @@ UniValue listtransactions(const UniValue& params, bool fHelp, const CPubKey& myp
|
||||
UniValue ret(UniValue::VARR);
|
||||
|
||||
std::list<CAccountingEntry> acentries;
|
||||
CWallet::TxItems txOrdered = pwalletMain->OrderedTxItems(acentries, strAccount);
|
||||
CWallet::TxItems txOrdered = pwalletMain->OrderedTxItems(acentries, strAccount, true);
|
||||
|
||||
// iterate backwards until we have nCount items to return:
|
||||
// iterate backwards until we have nCount items to return. The first nFrom entries are only
|
||||
// counted (nSkipLeft), never built, since they are not returned.
|
||||
int nSkipLeft = nFrom;
|
||||
for (CWallet::TxItems::reverse_iterator it = txOrdered.rbegin(); it != txOrdered.rend(); ++it)
|
||||
{
|
||||
CWalletTx *const pwtx = (*it).second.first;
|
||||
if (pwtx != 0)
|
||||
{
|
||||
ListTransactions(*pwtx, strAccount, 0, true, ret, filter);
|
||||
ListTransactions(*pwtx, strAccount, 0, true, ret, filter, &nSkipLeft);
|
||||
}
|
||||
CAccountingEntry *const pacentry = (*it).second.second;
|
||||
if (pacentry != 0)
|
||||
AcentryToJSON(*pacentry, strAccount, ret);
|
||||
AcentryToJSON(*pacentry, strAccount, ret, &nSkipLeft);
|
||||
|
||||
if ((int)ret.size() >= (nCount+nFrom)) break;
|
||||
// (skipped + built) is the entry count the loop used to test
|
||||
if ((nFrom - nSkipLeft) + (int)ret.size() >= (nCount+nFrom)) break;
|
||||
}
|
||||
// ret is newest to oldest
|
||||
// ret holds the entries after the first nSkipped ones, newest to oldest
|
||||
const int nSkipped = nFrom - nSkipLeft;
|
||||
const int nSeen = nSkipped + (int)ret.size();
|
||||
|
||||
if (nFrom > (int)ret.size())
|
||||
nFrom = ret.size();
|
||||
if ((nFrom + nCount) > (int)ret.size())
|
||||
nCount = ret.size() - nFrom;
|
||||
if (nFrom > nSeen)
|
||||
nFrom = nSeen;
|
||||
if ((nFrom + nCount) > nSeen)
|
||||
nCount = nSeen - nFrom;
|
||||
|
||||
vector<UniValue> arrTmp = ret.getValues();
|
||||
// The reply is entries [nFrom, nFrom+nCount) of the newest-first sequence, oldest to newest.
|
||||
// Entry k of that sequence is ret[k - nSkipped] (nFrom >= nSkipped holds here).
|
||||
UniValue result(UniValue::VARR);
|
||||
const std::vector<UniValue>& vNewestFirst = ret.getValues();
|
||||
for (int i = nFrom + nCount; i > nFrom; --i)
|
||||
result.push_back(vNewestFirst[i - 1 - nSkipped]);
|
||||
|
||||
vector<UniValue>::iterator first = arrTmp.begin();
|
||||
std::advance(first, nFrom);
|
||||
vector<UniValue>::iterator last = arrTmp.begin();
|
||||
std::advance(last, nFrom+nCount);
|
||||
|
||||
if (last != arrTmp.end()) arrTmp.erase(last, arrTmp.end());
|
||||
if (first != arrTmp.begin()) arrTmp.erase(arrTmp.begin(), first);
|
||||
|
||||
std::reverse(arrTmp.begin(), arrTmp.end()); // Return oldest to newest
|
||||
|
||||
ret.clear();
|
||||
ret.setArray();
|
||||
ret.push_backV(arrTmp);
|
||||
|
||||
return ret;
|
||||
return result;
|
||||
}
|
||||
|
||||
UniValue listaccounts(const UniValue& params, bool fHelp, const CPubKey& mypk)
|
||||
@@ -2238,7 +2266,7 @@ UniValue gettransaction(const UniValue& params, bool fHelp, const CPubKey& mypk)
|
||||
ListTransactions(wtx, "*", 0, false, details, filter);
|
||||
entry.push_back(Pair("details", details));
|
||||
|
||||
string strHex = EncodeHexTx(static_cast<CTransaction>(wtx));
|
||||
string strHex = EncodeHexTx(static_cast<const CTransaction&>(wtx));
|
||||
entry.push_back(Pair("hex", strHex));
|
||||
|
||||
return entry;
|
||||
@@ -2960,6 +2988,26 @@ UniValue listunspent(const UniValue& params, bool fHelp, const CPubKey& mypk)
|
||||
return results;
|
||||
}
|
||||
|
||||
// Wallet transactions ordered by nOrderPos without copying them (the old code copied every CWalletTx,
|
||||
// with every note's witness list, into a map<int64_t,CWalletTx>). The pointers point into
|
||||
// pwalletMain->mapWallet and are valid only while cs_wallet is held and nothing erases from mapWallet
|
||||
// (std::map insertions do not invalidate them). Reproduces map<int64_t,CWalletTx>::insert exactly:
|
||||
// ascending nOrderPos and, for equal nOrderPos, only the first tx in mapWallet (txid) order, because
|
||||
// insert() never replaces an existing key; hence stable_sort + unique.
|
||||
typedef std::pair<int64_t, const CWalletTx*> OrderedWtx;
|
||||
static bool OrderedWtxLess(const OrderedWtx& a, const OrderedWtx& b) { return a.first < b.first; }
|
||||
static bool OrderedWtxSamePos(const OrderedWtx& a, const OrderedWtx& b) { return a.first == b.first; }
|
||||
static void BuildOrderedWalletTxs(std::vector<OrderedWtx>& out)
|
||||
{
|
||||
AssertLockHeld(pwalletMain->cs_wallet);
|
||||
out.clear();
|
||||
out.reserve(pwalletMain->mapWallet.size());
|
||||
for (std::map<uint256, CWalletTx>::const_iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it)
|
||||
out.push_back(OrderedWtx(it->second.nOrderPos, &it->second));
|
||||
std::stable_sort(out.begin(), out.end(), OrderedWtxLess);
|
||||
out.erase(std::unique(out.begin(), out.end(), OrderedWtxSamePos), out.end());
|
||||
}
|
||||
|
||||
// Return current blockchain status, wallet balance, address balance and the last 200 transactions
|
||||
UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
{
|
||||
@@ -3031,14 +3079,14 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
addressBalances.insert(make_pair(addressString,txAmounts));
|
||||
}
|
||||
|
||||
//Create Ordered List
|
||||
// Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
|
||||
// As before, the first tx inserted for an nOrderPos wins.
|
||||
map<int64_t,const CWalletTx*> orderedTxs;
|
||||
// Per-call key/nullifier memo for the note loop below and for zsWalletTxJSON (rpchushwallet.h).
|
||||
// Declared after the LOCK2 above, so it is destroyed before cs_wallet is released.
|
||||
ZsTxJSONContext zsCtx;
|
||||
|
||||
//Classify every wallet tx for the balances (the ordered list for listtransactions is built below)
|
||||
for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
|
||||
const uint256& wtxid = it->first;
|
||||
const CWalletTx& wtx = (*it).second;
|
||||
orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
|
||||
|
||||
unsigned int txType = 0;
|
||||
// 0 Unassigend
|
||||
@@ -3107,18 +3155,19 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
|
||||
for (auto & pair : wtx.mapSaplingNoteData) {
|
||||
SaplingOutPoint op = pair.first;
|
||||
SaplingNoteData nd = pair.second;
|
||||
const SaplingNoteData& nd = pair.second;
|
||||
|
||||
//Skip Spent
|
||||
if (nd.nullifier && pwalletMain->IsSaplingSpent(*nd.nullifier))
|
||||
continue;
|
||||
|
||||
//Decrypt sapling incoming commitments using IVK
|
||||
for (auto addr : zs_addresses) {
|
||||
libzcash::SaplingExtendedSpendingKey extsk;
|
||||
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
|
||||
const std::vector<ZsTxJSONContext::Key>& zsKeys = zsCtx.keys();
|
||||
for (size_t k = 0; k < zsKeys.size(); k++) {
|
||||
const libzcash::SaplingPaymentAddress& addr = zsKeys[k].addr;
|
||||
{ // spending-key availability is resolved once per call, in ZsTxJSONContext::keys()
|
||||
auto pt = libzcash::SaplingNotePlaintext::decrypt(
|
||||
wtx.vShieldedOutput[op.n].encCiphertext,extsk.expsk.full_viewing_key().in_viewing_key(),wtx.vShieldedOutput[op.n].ephemeralKey,wtx.vShieldedOutput[op.n].cm);
|
||||
wtx.vShieldedOutput[op.n].encCiphertext,zsKeys[k].ivk,wtx.vShieldedOutput[op.n].ephemeralKey,wtx.vShieldedOutput[op.n].cm);
|
||||
|
||||
if (txType == 0 && pwalletMain->IsLockedNote(op))
|
||||
txType == 3;
|
||||
@@ -3230,11 +3279,14 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
}
|
||||
|
||||
|
||||
const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
|
||||
//Ordered list of pointers into mapWallet (cs_wallet is held for the whole call)
|
||||
std::vector<OrderedWtx> orderedTxs;
|
||||
BuildOrderedWalletTxs(orderedTxs);
|
||||
|
||||
uint64_t t = GetTime();
|
||||
for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it)
|
||||
for (std::vector<OrderedWtx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it)
|
||||
{
|
||||
const CWalletTx& wtx = *(*it).second;
|
||||
const CWalletTx& wtx = *(it->second);
|
||||
|
||||
if (!CheckFinalTx(wtx))
|
||||
continue;
|
||||
@@ -3251,7 +3303,7 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
continue;
|
||||
}
|
||||
|
||||
zsWalletTxJSON(wtx, trans, "*", false, 0, historyKeys);
|
||||
zsWalletTxJSON(wtx, trans, "*", false, 0, &zsCtx);
|
||||
if (trans.size() >= nCount) break;
|
||||
|
||||
}
|
||||
@@ -3517,21 +3569,16 @@ UniValue z_listreceivedaddress(const UniValue& params, bool fHelp,const CPubKey&
|
||||
if (nFilter < 0)
|
||||
throw runtime_error("Filter must be greater that 0.");
|
||||
|
||||
//Created Ordered Transaction Map
|
||||
// Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
|
||||
// As before, the first tx inserted for an nOrderPos wins.
|
||||
map<int64_t,const CWalletTx*> orderedTxs;
|
||||
for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
|
||||
const CWalletTx& wtx = (*it).second;
|
||||
orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
|
||||
}
|
||||
//Ordered list of pointers into mapWallet (cs_wallet is held for the whole call)
|
||||
std::vector<OrderedWtx> orderedTxs;
|
||||
BuildOrderedWalletTxs(orderedTxs);
|
||||
ZsTxJSONContext zsCtx; // per-call key/nullifier memo (rpchushwallet.h)
|
||||
|
||||
|
||||
const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
|
||||
uint64_t t = GetTime();
|
||||
//Reverse Iterate thru transactions
|
||||
for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
|
||||
const CWalletTx& wtx = *(*it).second;
|
||||
for (std::vector<OrderedWtx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
|
||||
const CWalletTx& wtx = *(it->second);
|
||||
bool includeTransaction = true;
|
||||
|
||||
//Excude transactions with less confirmations than required
|
||||
@@ -3552,7 +3599,7 @@ UniValue z_listreceivedaddress(const UniValue& params, bool fHelp,const CPubKey&
|
||||
}
|
||||
|
||||
if (includeTransaction) {
|
||||
zsWalletTxJSON(wtx, ret, params[0].get_str() , true, 2, historyKeys);
|
||||
zsWalletTxJSON(wtx, ret, params[0].get_str() , true, 2, &zsCtx);
|
||||
}
|
||||
|
||||
if (ret.size() > nCount) break;
|
||||
@@ -3603,7 +3650,7 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
if (!EnsureWalletIsAvailable(fHelp))
|
||||
return NullUniValue;
|
||||
|
||||
if (fHelp || params.size() > 5 || params.size() == 3)
|
||||
if (fHelp || params.size() > 6 || params.size() == 3)
|
||||
throw runtime_error(
|
||||
"z_listsentbyaddress\n"
|
||||
"\nReturns decrypted DragonX outputs sent to a single address.\n"
|
||||
@@ -3628,11 +3675,16 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
"5. \"Count:\" (numeric, optional, default=9999999) \n"
|
||||
" Last n number of transactions returned\n"
|
||||
"\n"
|
||||
"6. \"Sends only:\" (boolean, optional, default=false) \n"
|
||||
" true: omit the \"spends\" and \"received\" keys of every entry and skip the\n"
|
||||
" trial decryption they need. false: the full historic output.\n"
|
||||
"\n"
|
||||
"Default Parameters:\n"
|
||||
"2. 0 - O confimations required\n"
|
||||
"3. 0 - Returns all transactions\n"
|
||||
"4. 9999999 - Ignored\n"
|
||||
"5. 9999999 - Return the last 9,999,999 transactions.\n"
|
||||
"6. false - Include the spends and received sections.\n"
|
||||
"\n"
|
||||
"\nResult:\n"
|
||||
" \"txid\": \"transactionid\", (string) The transaction id.\n"
|
||||
@@ -3681,6 +3733,7 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
int64_t nFilterType = 0;
|
||||
int64_t nFilter = 9999999;
|
||||
int64_t nCount = 9999999;
|
||||
bool fSendsOnly = false; // opt-in, see help: omits the "spends"/"received" keys
|
||||
|
||||
if (params.size() >= 2)
|
||||
nMinConfirms = params[1].get_int64();
|
||||
@@ -3690,10 +3743,14 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
nFilter = params[3].get_int64();
|
||||
}
|
||||
|
||||
if (params.size() == 5) {
|
||||
if (params.size() >= 5) {
|
||||
nCount = params[4].get_int64();
|
||||
}
|
||||
|
||||
if (params.size() == 6) {
|
||||
fSendsOnly = params[5].get_bool();
|
||||
}
|
||||
|
||||
if (nMinConfirms < 0)
|
||||
throw runtime_error("Minimum confimations must be greater that 0");
|
||||
|
||||
@@ -3703,21 +3760,16 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
if (nFilter < 0)
|
||||
throw runtime_error("Filter must be greater that 0.");
|
||||
|
||||
//Created Ordered Transaction Map
|
||||
// Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
|
||||
// As before, the first tx inserted for an nOrderPos wins.
|
||||
map<int64_t,const CWalletTx*> orderedTxs;
|
||||
for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
|
||||
const CWalletTx& wtx = (*it).second;
|
||||
orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
|
||||
}
|
||||
//Ordered list of pointers into mapWallet (cs_wallet is held for the whole call)
|
||||
std::vector<OrderedWtx> orderedTxs;
|
||||
BuildOrderedWalletTxs(orderedTxs);
|
||||
ZsTxJSONContext zsCtx; // per-call key/nullifier memo (rpchushwallet.h)
|
||||
|
||||
|
||||
const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
|
||||
uint64_t t = GetTime();
|
||||
//Reverse Iterate thru transactions
|
||||
for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
|
||||
const CWalletTx& wtx = *(*it).second;
|
||||
for (std::vector<OrderedWtx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
|
||||
const CWalletTx& wtx = *(it->second);
|
||||
|
||||
if (!CheckFinalTx(wtx))
|
||||
continue;
|
||||
@@ -3740,21 +3792,21 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
|
||||
if (nFilterType == 2 && wtx.GetDepthInMainChain() > nFilter)
|
||||
continue;
|
||||
|
||||
zsWalletTxJSON(wtx, ret, "*", false, 0, historyKeys);
|
||||
zsWalletTxJSON(wtx, ret, "*", false, fSendsOnly ? 3 : 0, &zsCtx);
|
||||
|
||||
|
||||
if (ret.size() >= nCount) break;
|
||||
}
|
||||
|
||||
vector<UniValue> arrTmp = ret.getValues();
|
||||
|
||||
std::reverse(arrTmp.begin(), arrTmp.end()); // Return oldest to newest
|
||||
|
||||
ret.clear();
|
||||
ret.setArray();
|
||||
ret.push_backV(arrTmp);
|
||||
|
||||
return ret;
|
||||
// Return oldest to newest (ret was filled newest first): one copy of each entry instead of
|
||||
// getValues() + std::reverse + push_backV
|
||||
UniValue oldestFirst(UniValue::VARR);
|
||||
{
|
||||
const std::vector<UniValue>& newestFirst = ret.getValues();
|
||||
for (std::vector<UniValue>::const_reverse_iterator rit = newestFirst.rbegin(); rit != newestFirst.rend(); ++rit)
|
||||
oldestFirst.push_back(*rit);
|
||||
}
|
||||
return oldestFirst;
|
||||
}
|
||||
|
||||
UniValue z_getstats(const UniValue& params, bool fHelp, const CPubKey& mypk)
|
||||
@@ -4318,7 +4370,7 @@ UniValue z_listunspent(const UniValue& params, bool fHelp, const CPubKey& mypk)
|
||||
for (auto & entry : saplingEntries) {
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
|
||||
int nHeight = HeightFromDepth(entry.confirmations);
|
||||
int nHeight = tx_height(entry.op.hash);
|
||||
int dpowconfs = hush_dpowconfs(nHeight, entry.confirmations);
|
||||
|
||||
// Only return notarized results when minconf>1
|
||||
@@ -4651,7 +4703,7 @@ UniValue z_listreceivedbyaddress(const UniValue& params, bool fHelp, const CPubK
|
||||
if (boost::get<libzcash::SaplingPaymentAddress>(&zaddr) != nullptr) {
|
||||
for (SaplingNoteEntry & entry : saplingEntries) {
|
||||
UniValue obj(UniValue::VOBJ);
|
||||
int nHeight = HeightFromDepth(entry.confirmations);
|
||||
int nHeight = tx_height(entry.op.hash);
|
||||
int dpowconfs = hush_dpowconfs(nHeight, entry.confirmations);
|
||||
// Only return notarized results when minconf>1
|
||||
if (nMinDepth > 1 && dpowconfs == 1)
|
||||
@@ -4671,7 +4723,7 @@ UniValue z_listreceivedbyaddress(const UniValue& params, bool fHelp, const CPubK
|
||||
}
|
||||
obj.push_back(Pair("outindex", (int)entry.op.n));
|
||||
obj.push_back(Pair("rawconfirmations", entry.confirmations));
|
||||
const CWalletTx& wtx = pwalletMain->mapWallet.at(entry.op.hash);
|
||||
const CWalletTx& wtx = pwalletMain->mapWallet.at(entry.op.hash); //.ToString());
|
||||
obj.push_back(Pair("time", wtx.GetTxTime()));
|
||||
|
||||
obj.push_back(Pair("confirmations", dpowconfs));
|
||||
@@ -4924,7 +4976,7 @@ UniValue z_viewtransaction(const UniValue& params, bool fHelp, const CPubKey& my
|
||||
continue;
|
||||
}
|
||||
auto op = res->second;
|
||||
auto wtxPrev = pwalletMain->mapWallet.at(op.hash);
|
||||
const CWalletTx& wtxPrev = pwalletMain->mapWallet.at(op.hash);
|
||||
auto decrypted = wtxPrev.DecryptSaplingNote(op).get();
|
||||
auto notePt = decrypted.first;
|
||||
auto pa = decrypted.second;
|
||||
@@ -5027,7 +5079,12 @@ UniValue z_getoperationresult(const UniValue& params, bool fHelp, const CPubKey&
|
||||
+ HelpExampleRpc("z_getoperationresult", "'[\"operationid\", ... ]'")
|
||||
);
|
||||
|
||||
// This call will remove finished operations
|
||||
// This call will remove finished operations. Unlike z_getoperationstatus it keeps cs_main +
|
||||
// cs_wallet: CWallet::RunSaplingSweep/RunSaplingConsolidation/RunAutoShieldCoinbase cancel and
|
||||
// remove the previous automated operation inside one cs_wallet critical section, and holding
|
||||
// cs_wallet here keeps this removal serialised with that and with other z_getoperationresult
|
||||
// callers, exactly as before.
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
return z_getoperationstatus_IMPL(params, true);
|
||||
}
|
||||
|
||||
@@ -5056,7 +5113,12 @@ UniValue z_getoperationstatus(const UniValue& params, bool fHelp, const CPubKey&
|
||||
|
||||
UniValue z_getoperationstatus_IMPL(const UniValue& params, bool fRemoveFinishedOperations=false)
|
||||
{
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
// No cs_main/cs_wallet here: nothing below reads wallet or chain state. AsyncRPCQueue guards
|
||||
// its operation map with its own mutex, and AsyncRPCOperation::getStatus() (and every
|
||||
// override) reads only the atomic state_, members fixed at construction (id_, creation_time_,
|
||||
// contextinfo_, targetHeight_) and result/error under the operation's own lock_. Holding the
|
||||
// wallet locks only made this poll wait behind wallet scans and block connection. The removing
|
||||
// form is serialised by its caller, z_getoperationresult, which still holds cs_main + cs_wallet.
|
||||
|
||||
std::set<AsyncRPCOperationId> filter;
|
||||
if (params.size()==1) {
|
||||
@@ -5067,11 +5129,18 @@ UniValue z_getoperationstatus_IMPL(const UniValue& params, bool fRemoveFinishedO
|
||||
}
|
||||
bool useFilter = (filter.size()>0);
|
||||
|
||||
UniValue ret(UniValue::VARR);
|
||||
std::shared_ptr<AsyncRPCQueue> q = getAsyncRPCQueue();
|
||||
std::vector<AsyncRPCOperationId> ids = q->getAllOperationIds();
|
||||
|
||||
for (auto id : ids) {
|
||||
// Status objects in enumeration order, and one (creation_time, index) key per object for the
|
||||
// sort. UniValue has no move operations, so sorting the objects themselves deep-copied them on
|
||||
// every comparison and every element move.
|
||||
std::vector<UniValue> objs;
|
||||
objs.reserve(ids.size());
|
||||
std::vector<std::pair<int64_t, size_t> > order;
|
||||
order.reserve(ids.size());
|
||||
|
||||
for (const AsyncRPCOperationId& id : ids) {
|
||||
if (useFilter && !filter.count(id))
|
||||
continue;
|
||||
|
||||
@@ -5087,26 +5156,32 @@ UniValue z_getoperationstatus_IMPL(const UniValue& params, bool fRemoveFinishedO
|
||||
if (fRemoveFinishedOperations) {
|
||||
// Caller is only interested in retrieving finished results
|
||||
if ("success"==s || "failed"==s || "cancelled"==s) {
|
||||
ret.push_back(obj);
|
||||
q->popOperationForId(id);
|
||||
objs.push_back(obj);
|
||||
// Report the operation only if this call is the one that removed it.
|
||||
if (q->popOperationForId(id)) {
|
||||
order.push_back(std::make_pair(operation->getCreationTime(), objs.size() - 1));
|
||||
} else {
|
||||
objs.pop_back();
|
||||
}
|
||||
}
|
||||
} else {
|
||||
ret.push_back(obj);
|
||||
objs.push_back(obj);
|
||||
order.push_back(std::make_pair(operation->getCreationTime(), objs.size() - 1));
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<UniValue> arrTmp = ret.getValues();
|
||||
|
||||
// sort results chronologically by creation_time
|
||||
std::sort(arrTmp.begin(), arrTmp.end(), [](UniValue a, UniValue b) -> bool {
|
||||
const int64_t t1 = find_value(a.get_obj(), "creation_time").get_int64();
|
||||
const int64_t t2 = find_value(b.get_obj(), "creation_time").get_int64();
|
||||
return t1 < t2;
|
||||
// sort results chronologically by creation_time. Compare creation_time only, never the index:
|
||||
// std::sort then makes exactly the comparisons and moves it made on the status objects before,
|
||||
// so operations created in the same second keep the order earlier releases returned.
|
||||
std::sort(order.begin(), order.end(),
|
||||
[](const std::pair<int64_t, size_t>& a, const std::pair<int64_t, size_t>& b) -> bool {
|
||||
return a.first < b.first;
|
||||
});
|
||||
|
||||
ret.clear();
|
||||
ret.setArray();
|
||||
ret.push_backV(arrTmp);
|
||||
UniValue ret(UniValue::VARR);
|
||||
for (size_t i = 0; i < order.size(); i++) {
|
||||
ret.push_back(objs[order[i].second]);
|
||||
}
|
||||
|
||||
return ret;
|
||||
}
|
||||
@@ -5385,7 +5460,7 @@ UniValue z_sendmany(const UniValue& params, bool fHelp, const CPubKey& mypk)
|
||||
// sort notes from largest to smallest, which means
|
||||
// we will spend the largest first
|
||||
std::sort(saplingEntries.begin(), saplingEntries.end(),
|
||||
[](const SaplingNoteEntry& i, const SaplingNoteEntry& j) -> bool {
|
||||
[](SaplingNoteEntry i, SaplingNoteEntry j) -> bool {
|
||||
return i.note.value() > j.note.value();
|
||||
});
|
||||
|
||||
@@ -6193,8 +6268,8 @@ UniValue z_listoperationids(const UniValue& params, bool fHelp, const CPubKey& m
|
||||
+ HelpExampleRpc("z_listoperationids", "")
|
||||
);
|
||||
|
||||
LOCK2(cs_main, pwalletMain->cs_wallet);
|
||||
|
||||
// No cs_main/cs_wallet: like z_getoperationstatus this only reads the self-locking async
|
||||
// operation queue and each operation's atomic state.
|
||||
std::string filter;
|
||||
bool useFilter = false;
|
||||
if (params.size()==1) {
|
||||
|
||||
+182
-411
@@ -43,9 +43,7 @@
|
||||
#include "wallet/asyncrpcoperation_autoshieldcoinbase.h"
|
||||
#include <random>
|
||||
#include <limits>
|
||||
#include <system_error>
|
||||
#include <thread>
|
||||
#include <chrono>
|
||||
#include <atomic>
|
||||
#include <mutex>
|
||||
#include "zcash/zip32.h"
|
||||
@@ -77,8 +75,8 @@ bool fPayAtLeastCustomFee = true;
|
||||
CBlockIndex *hush_chainactive(int32_t height);
|
||||
extern int32_t HUSH_LOADINGBLOCKS;
|
||||
int32_t hush_dpowconfs(int32_t height,int32_t numconfs);
|
||||
int tx_height( const uint256 &hash );
|
||||
bool fTxDeleteEnabled = false;
|
||||
bool fCheckNoteCache = false;
|
||||
bool fTxConflictDeleteEnabled = false;
|
||||
int fDeleteInterval = DEFAULT_TX_DELETE_INTERVAL;
|
||||
unsigned int fDeleteTransactionsAfterNBlocks = DEFAULT_TX_RETENTION_BLOCKS;
|
||||
@@ -577,23 +575,6 @@ void CWallet::ChainTip(const CBlockIndex *pindex,
|
||||
DecrementNoteWitnesses(pindex);
|
||||
UpdateNullifierNoteMapForBlock(pblock);
|
||||
}
|
||||
|
||||
// Track the block the wallet's state reflects, advancing only along blocks delivered in order:
|
||||
// a gap (the startup rescan was interrupted, so the wallet sits at its checkpoint behind the
|
||||
// notifier) pins the pointer, and SetBestChain keeps stamping the checkpoint.
|
||||
LOCK(cs_wallet);
|
||||
if (added) {
|
||||
if (pindexWalletTip == nullptr || pindex->pprev == pindexWalletTip)
|
||||
pindexWalletTip = pindex;
|
||||
} else if (pindex == pindexWalletTip) {
|
||||
pindexWalletTip = pindex->pprev;
|
||||
}
|
||||
}
|
||||
|
||||
void CWallet::SetWalletTip(const CBlockIndex* pindex)
|
||||
{
|
||||
LOCK(cs_wallet);
|
||||
pindexWalletTip = pindex;
|
||||
}
|
||||
|
||||
void CWallet::RunSaplingSweep(int blockHeight) {
|
||||
@@ -820,22 +801,10 @@ bool CWallet::CommitAutomatedTx(const CTransaction& tx) {
|
||||
|
||||
void CWallet::SetBestChain(const CBlockLocator& loc)
|
||||
{
|
||||
// FlushStateToDisk passes the chain tip, but ThreadNotifyWallets delivers blocks to the wallet
|
||||
// asynchronously and can lag it. Stamping a locator ahead of what the wallet has processed makes
|
||||
// the next start skip those blocks for good (no rescan, no notification): notes received in them
|
||||
// go missing and spends in them go unseen. So stamp the block the wallet's state reflects (seeded
|
||||
// by init before the notifier starts, then advanced by ChainTip). Init's own calls, made before
|
||||
// it is seeded, pass the right locator themselves.
|
||||
CBlockLocator locator = loc;
|
||||
{
|
||||
LOCK2(cs_main, cs_wallet);
|
||||
if (pindexWalletTip != nullptr)
|
||||
locator = chainActive.GetLocator(pindexWalletTip);
|
||||
}
|
||||
// Default ctor => fFlushOnClose=true => ~CDB runs a full BDB txn_checkpoint over the entire
|
||||
// cache. Fine for the hourly/shutdown callers; ruinous inside a scan (see SetBestChainNoFlush).
|
||||
CWalletDB walletdb(strWalletFile);
|
||||
SetBestChainINTERNAL(walletdb, locator);
|
||||
SetBestChainINTERNAL(walletdb, loc);
|
||||
}
|
||||
|
||||
bool CWallet::SetBestChainNoFlush(const CBlockLocator& loc)
|
||||
@@ -1263,11 +1232,9 @@ void ClearSingleNoteWitnessCache(NoteData* nd)
|
||||
nd->witnessRootValidated = false;
|
||||
}
|
||||
|
||||
// A note without a nullifier is never extended by BuildWitnessCache (its pending-note gate requires one),
|
||||
// so it doesn't pull the rebuild's start height down.
|
||||
int CWallet::SaplingWitnessMinimumHeight(const boost::optional<uint256>& nullifier, int nWitnessHeight, int nMinimumHeight)
|
||||
int CWallet::SaplingWitnessMinimumHeight(const uint256& nullifier, int nWitnessHeight, int nMinimumHeight)
|
||||
{
|
||||
if (nullifier && GetSaplingSpendDepth(*nullifier) <= WITNESS_CACHE_SIZE) {
|
||||
if (GetSaplingSpendDepth(nullifier) <= WITNESS_CACHE_SIZE) {
|
||||
nMinimumHeight = min(nWitnessHeight, nMinimumHeight);
|
||||
}
|
||||
return nMinimumHeight;
|
||||
@@ -1299,16 +1266,15 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
|
||||
uint256 blockRoot;
|
||||
uint256 witnessRoot;
|
||||
|
||||
//A note found with an incoming viewing key only (z_importviewingkey) can't be spent, so it needs
|
||||
//no witness, and it never gets a nullifier: deriving one takes the full viewing key, which
|
||||
//UpdateSaplingNullifierNoteMapWithTx skips without. Seeding it here meant clearing and reseeding
|
||||
//it from disk on every call (the no-nullifier rule below), then aborting on the missing nullifier
|
||||
//the first time it was confirmed. If its spending key is imported later, the rule below seeds it.
|
||||
if (!HaveSaplingFullViewingKey(nd->ivk))
|
||||
continue;
|
||||
|
||||
if (!nd->nullifier)
|
||||
if (!nd->nullifier) {
|
||||
::ClearSingleNoteWitnessCache(nd);
|
||||
// A note found with only an incoming viewing key (z_importviewingkey) never gets a
|
||||
// nullifier (UpdateSaplingNullifierNoteMapWithTx needs the full viewing key) and can't be
|
||||
// spent, so it needs no witness. Building one re-read its block on every call and then
|
||||
// aborted on *item.second.nullifier below.
|
||||
if (!HaveSaplingFullViewingKey(nd->ivk))
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!nd->witnesses.empty() && nd->witnessHeight > 0) {
|
||||
|
||||
@@ -1318,7 +1284,7 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
|
||||
|
||||
//Skip Validation when witness root has been validated
|
||||
if (nd->witnessRootValidated) {
|
||||
nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
|
||||
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -1330,12 +1296,12 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
|
||||
CBlockIndex* whIndex = chainActive[nd->witnessHeight];
|
||||
if (whIndex == NULL) {
|
||||
//witnessHeight strictly above the active chain (transient catch-up): cannot validate yet
|
||||
nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
|
||||
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
|
||||
continue;
|
||||
}
|
||||
if (nd->witnesses.front().root() == whIndex->hashFinalSaplingRoot) {
|
||||
nd->witnessRootValidated = true;
|
||||
nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
|
||||
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
|
||||
continue;
|
||||
}
|
||||
//root mismatch on the active chain -> desynced; fall through to rebuild below
|
||||
@@ -1347,7 +1313,7 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
|
||||
blockRoot = pblockindex->hashFinalSaplingRoot;
|
||||
if (witnessRoot == blockRoot) {
|
||||
nd->witnessRootValidated = true;
|
||||
nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
|
||||
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
@@ -1399,7 +1365,10 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
|
||||
}
|
||||
nd->witnessHeight = pblockindex->GetHeight();
|
||||
UpdateSaplingNullifierNoteMapWithTx(wtxItem.second);
|
||||
nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
|
||||
if (item.second.nullifier)
|
||||
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
|
||||
else
|
||||
LogPrintf("%s: no witness found for note %s:%d in its block; it stays unspendable until a rescan\n", __func__, wtxHash.ToString(), op.n);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1518,25 +1487,17 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
|
||||
work.reserve(pending.size());
|
||||
for (const PendingNote& p : pending) work.push_back(p.nd);
|
||||
|
||||
// Only a substantial bulk rebuild (e.g. a one-time post-upgrade repair) is logged and has its
|
||||
// anchors checked; routine 1-block tip extension stays quiet to avoid per-block log spam.
|
||||
// Only a substantial bulk rebuild (e.g. a one-time post-upgrade repair) is worth parallelizing
|
||||
// and logging; routine 1-block tip extension runs serially to avoid per-block thread-spawn
|
||||
// overhead and log spam.
|
||||
const bool bulkRebuild = (rangeLen > 100);
|
||||
|
||||
// Parallelize whenever the extension itself is expensive, not only for a bulk range: a routine
|
||||
// 1-block extension appends each of the block's commitments to every lagging note, so a large
|
||||
// wallet and a busy block (13k notes x 20 commitments) is seconds of hashing under
|
||||
// cs_main/cs_wallet. Spawning the threads costs ~1ms, so only small jobs stay serial.
|
||||
uint64_t nCms = 0;
|
||||
for (const std::vector<uint256>& cms : blockCms)
|
||||
nCms += cms.size();
|
||||
const bool fParallel = bulkRebuild || (uint64_t)work.size() * nCms >= 2000;
|
||||
|
||||
int nPar = (int)GetArg("-witnessbuildthreads", 0);
|
||||
if (nPar <= 0) nPar = (int)std::thread::hardware_concurrency();
|
||||
if (nPar <= 0) nPar = 1;
|
||||
if (nPar > (int)work.size()) nPar = (int)work.size();
|
||||
if (nPar < 1) nPar = 1;
|
||||
if (!fParallel) nPar = 1;
|
||||
if (!bulkRebuild) nPar = 1;
|
||||
|
||||
std::atomic<bool> failed(false);
|
||||
std::mutex failMtx;
|
||||
@@ -1548,27 +1509,14 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
|
||||
// clone-every-block path for A/B verification.
|
||||
bool fastRebuild = GetBoolArg("-witnessfastrebuild", true);
|
||||
|
||||
// A bulk rebuild reports how many notes failed to reach the canonical anchor. root() walks the
|
||||
// full depth-32 path (~2.4ms), so the check runs here on the workers and never on the routine
|
||||
// per-block path, where it used to hold cs_main/cs_wallet for ~30s per block on a ~13k-note
|
||||
// wallet purely to feed a log line that path never prints.
|
||||
const uint256 canonicalRoot = pindex->hashFinalSaplingRoot;
|
||||
std::atomic<int> nUnrecoverable(0);
|
||||
auto checkRoot = [&](const SaplingNoteData* nd) {
|
||||
if (bulkRebuild && (nd->witnesses.empty() || nd->witnesses.front().root() != canonicalRoot))
|
||||
nUnrecoverable++;
|
||||
};
|
||||
|
||||
auto worker = [&](int tid) {
|
||||
try {
|
||||
for (size_t k = (size_t)tid; k < work.size(); k += (size_t)nPar) {
|
||||
SaplingNoteData* nd = work[k];
|
||||
int startH = nd->witnessHeight;
|
||||
int nBlocks = tipHeight - startH;
|
||||
if (nBlocks <= 0) {
|
||||
checkRoot(nd);
|
||||
if (nBlocks <= 0)
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!fastRebuild || nBlocks <= CACHE) {
|
||||
// Reference / shallow path: clone every block (preserves pre-existing older snapshots).
|
||||
@@ -1609,7 +1557,6 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
|
||||
nd->witnesses.pop_back();
|
||||
nd->witnessHeight = tipHeight;
|
||||
}
|
||||
checkRoot(nd);
|
||||
}
|
||||
} catch (const std::exception& e) {
|
||||
std::lock_guard<std::mutex> lk(failMtx);
|
||||
@@ -1624,18 +1571,9 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
|
||||
if (nPar <= 1) {
|
||||
worker(0);
|
||||
} else {
|
||||
// If the OS refuses a thread (task limit, memory), run the remaining shards here instead of
|
||||
// letting the exception unwind past joinable threads (std::terminate). Each shard still runs
|
||||
// exactly once, so the result is unchanged.
|
||||
std::vector<std::thread> threads;
|
||||
threads.reserve(nPar);
|
||||
int t = 0;
|
||||
try {
|
||||
for (; t < nPar; t++) threads.emplace_back(worker, t);
|
||||
} catch (const std::system_error& e) {
|
||||
LogPrintf("%s: could only start %d of %d witness threads (%s); running the rest inline\n", __func__, t, nPar, e.what());
|
||||
}
|
||||
for (int u = t; u < nPar; u++) worker(u);
|
||||
for (int t = 0; t < nPar; t++) threads.emplace_back(worker, t);
|
||||
for (std::thread& th : threads) th.join();
|
||||
}
|
||||
int64_t tExtend = GetTimeMicros() - e0;
|
||||
@@ -1650,16 +1588,20 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
|
||||
// is unrecoverable here: it is left flagged (the GetSaplingNoteWitnesses majority-anchor guard
|
||||
// skips it for spends) and reported, rather than spun on indefinitely. witnessRootValidated is
|
||||
// in-memory only, so a fresh validation pass still runs on each restart and after any reorg
|
||||
// (DecrementNoteWitnesses clears it), keeping the heal self-correcting. The unrecoverable count
|
||||
// itself comes from checkRoot() on the workers, and only for a bulk rebuild.
|
||||
// (DecrementNoteWitnesses clears it), keeping the heal self-correcting.
|
||||
if (!work.empty()) {
|
||||
for (SaplingNoteData* nd : work)
|
||||
const uint256& canonicalRoot = pindex->hashFinalSaplingRoot;
|
||||
int nUnrecoverable = 0;
|
||||
for (SaplingNoteData* nd : work) {
|
||||
if (nd->witnesses.empty() || nd->witnesses.front().root() != canonicalRoot)
|
||||
nUnrecoverable++;
|
||||
nd->witnessRootValidated = true;
|
||||
}
|
||||
if (bulkRebuild) {
|
||||
LogPrintf("%s: rebuilt %u note witness cache(s) to height %d in %ldms using %d thread(s)%s\n",
|
||||
__func__, (unsigned)work.size(), tipHeight, (long)((tRead + tExtend) / 1000), nPar,
|
||||
nUnrecoverable.load()
|
||||
? strprintf(" [WARNING: %d note(s) could not be rebuilt to the canonical anchor and were skipped]", nUnrecoverable.load()).c_str()
|
||||
nUnrecoverable
|
||||
? strprintf(" [WARNING: %d note(s) could not be rebuilt to the canonical anchor and were skipped]", nUnrecoverable).c_str()
|
||||
: "");
|
||||
}
|
||||
}
|
||||
@@ -1769,10 +1711,12 @@ int64_t CWallet::IncOrderPosNext(CWalletDB *pwalletdb)
|
||||
return nRet;
|
||||
}
|
||||
|
||||
CWallet::TxItems CWallet::OrderedTxItems(std::list<CAccountingEntry>& acentries, std::string strAccount)
|
||||
CWallet::TxItems CWallet::OrderedTxItems(std::list<CAccountingEntry>& acentries, std::string strAccount, bool fReadOnlyDb)
|
||||
{
|
||||
AssertLockHeld(cs_wallet); // mapWallet
|
||||
CWalletDB walletdb(strWalletFile);
|
||||
// Only a read cursor over the accounting entries runs on this handle. Pure readers open it
|
||||
// read-only and without flush-on-close, so the call does not end in a txn_checkpoint.
|
||||
CWalletDB walletdb(strWalletFile, fReadOnlyDb ? "r" : "r+", !fReadOnlyDb);
|
||||
|
||||
// First: get all CWalletTx and CAccountingEntry into a sorted-by-order multimap.
|
||||
TxItems txOrdered;
|
||||
@@ -1981,8 +1925,6 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
|
||||
bool fUpdated = false;
|
||||
if (!fInsertedNew)
|
||||
{
|
||||
// Re-mined in a different block (a reorg): its notes sit at new tree positions.
|
||||
bool fSameBlock = wtx.hashBlock.IsNull() || wtxIn.hashBlock.IsNull() || wtxIn.hashBlock == wtx.hashBlock;
|
||||
// Merge
|
||||
if (!wtxIn.hashBlock.IsNull() && wtxIn.hashBlock != wtx.hashBlock)
|
||||
{
|
||||
@@ -1995,7 +1937,7 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
|
||||
wtx.nIndex = wtxIn.nIndex;
|
||||
fUpdated = true;
|
||||
}
|
||||
if (UpdatedNoteData(wtxIn, wtx, fSameBlock)) {
|
||||
if (UpdatedNoteData(wtxIn, wtx)) {
|
||||
fUpdated = true;
|
||||
}
|
||||
if (wtxIn.fFromMe && wtxIn.fFromMe != wtx.fFromMe)
|
||||
@@ -2032,92 +1974,29 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
|
||||
return true;
|
||||
}
|
||||
|
||||
// True if both maps hold the same notes with identical persisted state (every field SaplingNoteData
|
||||
// serializes). SaplingNoteData::operator== ignores the witnesses, so it cannot tell a real change
|
||||
// from a redundant update.
|
||||
// Copies the decrypted plaintext of each incoming note into the matching (same outpoint and ivk) entry that
|
||||
// lacks one. UpdatedNoteData uses it when it keeps the wallet's entries: a replay or rescan has just
|
||||
// decrypted the notes (FindMySaplingNotes), and entries loaded from wallet.dat have no plaintext. The cache
|
||||
// is in memory only and outside every comparison, so this never makes a tx look changed.
|
||||
static void KeepNotePlaintexts(const mapSaplingNoteData_t& incoming, mapSaplingNoteData_t& wallet)
|
||||
{
|
||||
for (const mapSaplingNoteData_t::value_type& in : incoming) {
|
||||
if (!in.second.cachedPlaintext)
|
||||
continue;
|
||||
mapSaplingNoteData_t::iterator it = wallet.find(in.first);
|
||||
if (it != wallet.end() && !it->second.cachedPlaintext && it->second.ivk == in.second.ivk)
|
||||
it->second.cachedPlaintext = in.second.cachedPlaintext;
|
||||
}
|
||||
}
|
||||
|
||||
static bool SaplingNoteDataIdentical(const mapSaplingNoteData_t& a, const mapSaplingNoteData_t& b)
|
||||
{
|
||||
if (a.size() != b.size())
|
||||
return false;
|
||||
for (const mapSaplingNoteData_t::value_type& item : a) {
|
||||
mapSaplingNoteData_t::const_iterator it = b.find(item.first);
|
||||
if (it == b.end())
|
||||
return false;
|
||||
const SaplingNoteData& x = item.second;
|
||||
const SaplingNoteData& y = it->second;
|
||||
if (!(x.ivk == y.ivk) || x.nullifier != y.nullifier || x.witnessHeight != y.witnessHeight || x.witnesses != y.witnesses)
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool CWallet::UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock)
|
||||
bool CWallet::UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx)
|
||||
{
|
||||
bool unchangedSaplingFlag = (wtxIn.mapSaplingNoteData.empty() || wtxIn.mapSaplingNoteData == wtx.mapSaplingNoteData);
|
||||
if (unchangedSaplingFlag) {
|
||||
KeepNotePlaintexts(wtxIn.mapSaplingNoteData, wtx.mapSaplingNoteData);
|
||||
return false;
|
||||
}
|
||||
if (!unchangedSaplingFlag) {
|
||||
auto tmp = wtxIn.mapSaplingNoteData;
|
||||
// Ensure we keep any cached witnesses we may already have
|
||||
|
||||
auto tmp = wtxIn.mapSaplingNoteData;
|
||||
// Ensure we keep any cached witnesses we may already have
|
||||
|
||||
for (const mapSaplingNoteData_t::value_type& nd : wtx.mapSaplingNoteData) {
|
||||
if (tmp.count(nd.first) && nd.second.witnesses.size() > 0) {
|
||||
tmp.at(nd.first).witnesses.assign(
|
||||
nd.second.witnesses.cbegin(), nd.second.witnesses.cend());
|
||||
//PART B2: only carry over witnessHeight TOGETHER with the witnesses it describes.
|
||||
//Copying it unconditionally (when witnesses are NOT copied) advances the height past
|
||||
//the actual witness state for a whole tx's notes at once = the batch desync originator.
|
||||
tmp.at(nd.first).witnessHeight = nd.second.witnessHeight;
|
||||
//The nullifier is derived from the note and its witness position, so it travels with the
|
||||
//witnesses too. wtxIn comes from FindMySaplingNotes, which never sets it; dropping it made
|
||||
//VerifyAndSetInitialWitness clear and reseed the witnesses just carried over, so replaying
|
||||
//or rescanning already-witnessed txs forced a rebuild of every one of their notes. A tx
|
||||
//re-mined in another block is the exception: its notes moved, so the old nullifier is
|
||||
//wrong and dropping it still reseeds them at the new position.
|
||||
if (nd.second.nullifier && fSameBlock)
|
||||
tmp.at(nd.first).nullifier = nd.second.nullifier;
|
||||
for (const std::pair <SaplingOutPoint, SaplingNoteData> nd : wtx.mapSaplingNoteData) {
|
||||
if (tmp.count(nd.first) && nd.second.witnesses.size() > 0) {
|
||||
tmp.at(nd.first).witnesses.assign(
|
||||
nd.second.witnesses.cbegin(), nd.second.witnesses.cend());
|
||||
//PART B2: only carry over witnessHeight TOGETHER with the witnesses it describes.
|
||||
//Copying it unconditionally (when witnesses are NOT copied) advances the height past
|
||||
//the actual witness state for a whole tx's notes at once = the batch desync originator.
|
||||
tmp.at(nd.first).witnessHeight = nd.second.witnessHeight;
|
||||
}
|
||||
}
|
||||
|
||||
// Now copy over the updated note data
|
||||
wtx.mapSaplingNoteData = tmp;
|
||||
}
|
||||
|
||||
// Keep a decrypted plaintext the wallet already holds (in memory only; a pure function of the
|
||||
// outpoint and ivk) when the incoming note data lacks one.
|
||||
for (const mapSaplingNoteData_t::value_type& nd : wtx.mapSaplingNoteData) {
|
||||
mapSaplingNoteData_t::iterator it = tmp.find(nd.first);
|
||||
if (it != tmp.end() && !it->second.cachedPlaintext && nd.second.cachedPlaintext && it->second.ivk == nd.second.ivk)
|
||||
it->second.cachedPlaintext = nd.second.cachedPlaintext;
|
||||
}
|
||||
|
||||
// A replayed or rescanned tx usually carries nothing new; don't rewrite it to disk. Do have
|
||||
// VerifyAndSetInitialWitness re-check the kept witnesses against the active chain once (the
|
||||
// flag is in memory only): the chain may have moved under them, and it keeps a rescan able to
|
||||
// find and reseed a witness that went bad, without rebuilding the healthy ones.
|
||||
if (SaplingNoteDataIdentical(tmp, wtx.mapSaplingNoteData)) {
|
||||
for (mapSaplingNoteData_t::value_type& item : wtx.mapSaplingNoteData)
|
||||
item.second.witnessRootValidated = false;
|
||||
KeepNotePlaintexts(tmp, wtx.mapSaplingNoteData);
|
||||
return false;
|
||||
}
|
||||
|
||||
// Now copy over the updated note data
|
||||
wtx.mapSaplingNoteData = tmp;
|
||||
return true;
|
||||
return !unchangedSaplingFlag;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -2290,24 +2169,21 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
|
||||
|
||||
// Protocol Spec: 4.19 Block Chain Scanning (Sapling)
|
||||
for (uint32_t i = 0; i < nZouts; ++i) {
|
||||
const OutputDescription& output = tx.vShieldedOutput[i];
|
||||
const OutputDescription output = tx.vShieldedOutput[i];
|
||||
bool found = false;
|
||||
for (auto it = mapSaplingFullViewingKeys.begin(); it != mapSaplingFullViewingKeys.end(); ++it) {
|
||||
SaplingIncomingViewingKey ivk = it->first;
|
||||
auto result = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
|
||||
auto result = SaplingNotePlaintext::decrypt(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
|
||||
if (result) {
|
||||
// The note's (d, pk_d) is the address it was sent to, exactly what ivk.address(d) would
|
||||
// derive again: decrypt_with_note already computed pk_d from this ivk and d.
|
||||
const libzcash::SaplingPaymentAddress address(result->second.d, result->second.pk_d);
|
||||
if (mapSaplingIncomingViewingKeys.count(address) == 0) {
|
||||
viewingKeysToAdd[address] = ivk;
|
||||
auto address = ivk.address(result.get().d);
|
||||
if (address && mapSaplingIncomingViewingKeys.count(address.get()) == 0) {
|
||||
viewingKeysToAdd[address.get()] = ivk;
|
||||
}
|
||||
// We don't cache the nullifier here as computing it requires knowledge of the note position
|
||||
// in the commitment tree, which can only be determined when the transaction has been mined.
|
||||
SaplingOutPoint op {hash, i};
|
||||
SaplingNoteData nd;
|
||||
nd.ivk = ivk;
|
||||
nd.cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
|
||||
noteData.insert(std::make_pair(op, nd));
|
||||
found = true;
|
||||
break;
|
||||
@@ -2316,7 +2192,7 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
|
||||
if (!found) {
|
||||
for (auto it = mapSaplingIncomingViewingKeys.begin(); it != mapSaplingIncomingViewingKeys.end(); ++it) {
|
||||
SaplingIncomingViewingKey ivk = it-> second;
|
||||
auto result = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
|
||||
auto result = SaplingNotePlaintext::decrypt(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
|
||||
if (!result) {
|
||||
continue;
|
||||
}
|
||||
@@ -2325,7 +2201,6 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
|
||||
SaplingOutPoint op {hash, i};
|
||||
SaplingNoteData nd;
|
||||
nd.ivk = ivk;
|
||||
nd.cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
|
||||
noteData.insert(std::make_pair(op, nd));
|
||||
break;
|
||||
}
|
||||
@@ -3072,12 +2947,13 @@ boost::optional<std::pair<
|
||||
SaplingPaymentAddress>> CWalletTx::DecryptSaplingNote(SaplingOutPoint op) const
|
||||
{
|
||||
// Check whether we can decrypt this SaplingOutPoint
|
||||
if (this->mapSaplingNoteData.count(op) == 0) {
|
||||
mapSaplingNoteData_t::const_iterator ndIt = this->mapSaplingNoteData.find(op);
|
||||
if (ndIt == this->mapSaplingNoteData.end()) {
|
||||
return boost::none;
|
||||
}
|
||||
|
||||
auto output = this->vShieldedOutput[op.n];
|
||||
auto nd = this->mapSaplingNoteData.at(op);
|
||||
const OutputDescription& output = this->vShieldedOutput[op.n];
|
||||
const SaplingNoteData& nd = ndIt->second;
|
||||
|
||||
auto maybe_pt = SaplingNotePlaintext::decrypt(
|
||||
output.encCiphertext,
|
||||
@@ -5746,137 +5622,6 @@ bool CMerkleTx::AcceptToMemoryPool(bool fLimitFree, bool fRejectAbsurdFee)
|
||||
return ::AcceptToMemoryPool(mempool, state, *this, fLimitFree, NULL, fRejectAbsurdFee);
|
||||
}
|
||||
|
||||
unsigned int CWallet::FillSaplingNoteCache(const std::vector<SaplingNoteToDecrypt>& notes)
|
||||
{
|
||||
AssertLockHeld(cs_wallet);
|
||||
|
||||
// Each decryption is independent: it reads its own output and writes only its own note's
|
||||
// cachedPlaintext. So a big batch (a whole wallet after a restart, ~0.5ms a note) is split across
|
||||
// threads the way BuildWitnessCache's Phase 2 is, while this thread keeps cs_wallet held. Workers
|
||||
// must not lock, or even assert a lock: under DEBUG_LOCKORDER a fresh thread has no lock stack.
|
||||
// Spawning costs ~1ms, so small batches stay serial.
|
||||
int nPar = 1;
|
||||
if (notes.size() >= 128) {
|
||||
nPar = (int)GetArg("-witnessbuildthreads", 0);
|
||||
if (nPar <= 0) nPar = (int)std::thread::hardware_concurrency();
|
||||
nPar = std::max(1, std::min(nPar, (int)(notes.size() / 64)));
|
||||
}
|
||||
|
||||
std::vector<unsigned int> filled(nPar, 0);
|
||||
auto worker = [&](int tid) {
|
||||
for (size_t k = (size_t)tid; k < notes.size(); k += (size_t)nPar) {
|
||||
const SaplingNoteToDecrypt& n = notes[k];
|
||||
try {
|
||||
auto result = SaplingNotePlaintext::decrypt_with_note(
|
||||
n.output->encCiphertext, n.nd->ivk, n.output->ephemeralKey, n.output->cm);
|
||||
if (result) {
|
||||
n.nd->cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
|
||||
filled[tid]++;
|
||||
}
|
||||
} catch (...) {
|
||||
// Left unfilled, exactly like a note that doesn't decrypt; callers handle that. Nothing may
|
||||
// escape a std::thread.
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
if (nPar <= 1) {
|
||||
worker(0);
|
||||
} else {
|
||||
// If the OS refuses a thread, run the remaining shards here instead of letting the exception
|
||||
// unwind past joinable threads (std::terminate). Each shard still runs exactly once.
|
||||
std::vector<std::thread> threads;
|
||||
threads.reserve(nPar);
|
||||
int t = 0;
|
||||
try {
|
||||
for (; t < nPar; t++) threads.emplace_back(worker, t);
|
||||
} catch (const std::exception& e) {
|
||||
LogPrintf("%s: could only start %d of %d threads (%s); decrypting the rest inline\n", __func__, t, nPar, e.what());
|
||||
}
|
||||
for (int u = t; u < nPar; u++) worker(u);
|
||||
for (std::thread& th : threads) th.join();
|
||||
}
|
||||
|
||||
unsigned int nFilled = 0;
|
||||
for (unsigned int f : filled)
|
||||
nFilled += f;
|
||||
nSaplingNoteCacheFills += nFilled;
|
||||
nSaplingNoteCacheFillThreads = nPar;
|
||||
LogPrint("bench", "%s: decrypted %u of %u notes using %d thread(s)\n", __func__, nFilled, (unsigned int)notes.size(), nPar);
|
||||
return nFilled;
|
||||
}
|
||||
|
||||
// wallet.dat doesn't store the plaintext, so without this the first balance call after a restart decrypts
|
||||
// the whole wallet under cs_main (~0.5ms a note). init runs it before RPC leaves warmup and before the
|
||||
// wallet notification thread starts; only the few RPCs allowed during warmup (key export, address
|
||||
// listing) can wait on cs_wallet meanwhile. Unlike GetFilteredNotes it never asserts, so a corrupt note
|
||||
// can't stop the node from starting (or from serving those RPCs); GetFilteredNotes still asserts on such
|
||||
// a note, as before.
|
||||
CWallet::SaplingNoteCacheWarmup CWallet::WarmSaplingNoteCache()
|
||||
{
|
||||
const std::chrono::steady_clock::time_point start = std::chrono::steady_clock::now();
|
||||
LOCK(cs_wallet);
|
||||
SaplingNoteCacheWarmup result;
|
||||
std::vector<SaplingNoteToDecrypt> uncached;
|
||||
for (auto& p : mapWallet) {
|
||||
CWalletTx& wtx = p.second;
|
||||
for (auto& item : wtx.mapSaplingNoteData) {
|
||||
result.notes++;
|
||||
if (item.second.cachedPlaintext)
|
||||
continue;
|
||||
if (item.first.n >= wtx.vShieldedOutput.size()) {
|
||||
LogPrint("zrpcunsafe", "%s: note %s:%u has no such output\n", __func__, item.first.hash.ToString(), item.first.n);
|
||||
result.noSuchOutput++;
|
||||
continue;
|
||||
}
|
||||
uncached.push_back(SaplingNoteToDecrypt { &item.second, &wtx.vShieldedOutput[item.first.n] });
|
||||
}
|
||||
}
|
||||
result.filled = uncached.empty() ? 0 : FillSaplingNoteCache(uncached);
|
||||
for (const SaplingNoteToDecrypt& n : uncached) {
|
||||
if (!n.nd->cachedPlaintext) {
|
||||
LogPrint("zrpcunsafe", "%s: a note of output %s does not decrypt under its ivk\n", __func__, n.output->cm.ToString());
|
||||
result.undecryptable++;
|
||||
}
|
||||
}
|
||||
const unsigned int nBad = result.noSuchOutput + result.undecryptable;
|
||||
const int64_t nMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - start).count();
|
||||
LogPrintf("%s: decrypted %u of %u Sapling notes in %dms%s\n", __func__, result.filled, result.notes, nMs,
|
||||
nBad ? strprintf("; %u could not be decrypted (-debug=zrpcunsafe lists them)", nBad) : "");
|
||||
return result;
|
||||
}
|
||||
|
||||
// -checknotecache: decrypts a cached note again, and logs and repairs any difference. GetFilteredNotes runs
|
||||
// it on every note that passes its tx, spent and locked filters, before the address filter, so a corrupt
|
||||
// cached address can't wrongly drop a note unchecked. A diagnostic for the cache: it costs the full
|
||||
// decryption the cache exists to avoid.
|
||||
static void CheckSaplingNoteCache(const SaplingOutPoint& op, SaplingNoteData& nd, const OutputDescription& output)
|
||||
{
|
||||
auto fresh = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, nd.ivk, output.ephemeralKey, output.cm);
|
||||
if (!fresh) {
|
||||
LogPrintf("%s: note %s:%u has a cached plaintext but no longer decrypts\n", __func__, op.hash.ToString(), op.n);
|
||||
return;
|
||||
}
|
||||
const SaplingNoteCache& cached = *nd.cachedPlaintext;
|
||||
const libzcash::SaplingNote& note = fresh->second;
|
||||
if (!(cached.note.d == note.d && cached.note.pk_d == note.pk_d && cached.note.r == note.r &&
|
||||
cached.note.value() == note.value() && cached.memo == fresh->first.memo())) {
|
||||
LogPrintf("%s: cached plaintext of note %s:%u differs from a fresh decryption; using the fresh one\n",
|
||||
__func__, op.hash.ToString(), op.n);
|
||||
nd.cachedPlaintext = SaplingNoteCache{ note, fresh->first.memo() };
|
||||
}
|
||||
}
|
||||
|
||||
// Replaces tx_height() on the wallet's note paths. tx_height() re-read the tx from disk through
|
||||
// GetTransaction, once per wallet tx (or per returned note) under LOCK2, and on a -txindex=0 node it
|
||||
// never found a fully shielded tx, so those notes were silently skipped. The depth already locates
|
||||
// the block: GetDepthInMainChain is chainActive.Height() - height + 1 for an active-chain tx.
|
||||
int HeightFromDepth(int nDepth)
|
||||
{
|
||||
AssertLockHeld(cs_main);
|
||||
return nDepth > 0 ? chainActive.Height() - nDepth + 1 : 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Find notes in the wallet filtered by payment address, min depth and ability to spend.
|
||||
* These notes are decrypted and added to the output parameter vector, outEntries.
|
||||
@@ -5897,6 +5642,26 @@ void CWallet::GetFilteredNotes(
|
||||
GetFilteredNotes(saplingEntries, filterAddresses, minDepth, INT_MAX, ignoreSpent, requireSpendingKey);
|
||||
}
|
||||
|
||||
namespace {
|
||||
|
||||
// GetFilteredNotes' decrypt-free address pre-filter is only built for at most this many
|
||||
// filter addresses. Verifying each one costs a Jubjub scalar multiplication, which buys
|
||||
// nothing when the filter is "every address in the wallet" (z_listunspent, z_getbalances).
|
||||
// Setting it to 0 disables the pre-filter entirely.
|
||||
const size_t GETFILTEREDNOTES_PREFILTER_MAX_ADDRESSES = 16;
|
||||
|
||||
// True if ivk < 2^251. A genuine Sapling ivk is CRH_ivk output with its top 5 bits cleared
|
||||
// (librustzcash_crh_ivk), so it is below 2^251 < r_J (the Jubjub subgroup order). Below r_J,
|
||||
// pk_d = [ivk] g_d determines ivk uniquely. librustzcash_ivk_to_pkd multiplies by the raw
|
||||
// 256-bit value without reducing it, so a value >= r_J aliases ivk - r_J. Byte 31 of the
|
||||
// little-endian representation is the most significant byte.
|
||||
bool IsCanonicalSaplingIvk(const libzcash::SaplingIncomingViewingKey& ivk)
|
||||
{
|
||||
return (ivk.begin()[31] & 0xF8) == 0;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
/**
|
||||
* Find notes in the wallet filtered by payment addresses, min depth, max depth,
|
||||
* if the note is spent, if a spending key is required, and if the notes are locked.
|
||||
@@ -5913,28 +5678,47 @@ void CWallet::GetFilteredNotes(
|
||||
{
|
||||
LOCK2(cs_main, cs_wallet);
|
||||
|
||||
// Every caller already holds LOCK2, so the lock above never waits and this times the walk alone.
|
||||
// Time spent waiting for the locks shows up in the RPC's own "took" line (rpc/server.cpp).
|
||||
const std::chrono::steady_clock::time_point walkStart = std::chrono::steady_clock::now();
|
||||
const size_t nEntriesBefore = saplingEntries.size();
|
||||
unsigned int nTxs = 0, nNotes = 0, nSpent = 0;
|
||||
LogPrintf("%s ignoreLocked=%d\n", __func__, ignoreLocked);
|
||||
|
||||
// Pass 1: the notes that pass every filter not needing the plaintext, in wallet order.
|
||||
struct Candidate {
|
||||
const SaplingOutPoint* op;
|
||||
SaplingNoteData* nd;
|
||||
const OutputDescription* output;
|
||||
int nDepth;
|
||||
};
|
||||
std::vector<Candidate> candidates;
|
||||
std::vector<SaplingNoteToDecrypt> uncached;
|
||||
for (auto & p : mapWallet) {
|
||||
// By reference: a copy deep-copies every note's witness list (up to WITNESS_CACHE_SIZE
|
||||
// witnesses each), and it was made before any filter ran. Not const: a missing plaintext is
|
||||
// cached onto the wallet's own entry.
|
||||
CWalletTx& wtx = p.second;
|
||||
// Decrypt-free address pre-filter. A note matches a Sapling filter address a only if
|
||||
// nd.ivk.address(a.d) == a. If a's recorded ivk r is canonical and provably generates a
|
||||
// (r.address(a.d) == a), then any other canonical ivk cannot generate a, so a note whose
|
||||
// canonical nd.ivk is not one of these r can be skipped without decrypting it. Recorded
|
||||
// pairs are not trusted as-is (z_importviewingkey stores any (ivk, zaddr) pair the user
|
||||
// supplies), so each one is verified. If any Sapling filter address has no recorded ivk,
|
||||
// a non-canonical one, or one that does not generate it, the pre-filter is off and every
|
||||
// note takes the old decrypt path. InvalidEncoding entries never equal a note's address.
|
||||
std::set<libzcash::SaplingIncomingViewingKey> prefilterIvks;
|
||||
bool usePrefilter = !filterAddresses.empty() &&
|
||||
filterAddresses.size() <= GETFILTEREDNOTES_PREFILTER_MAX_ADDRESSES;
|
||||
if (usePrefilter) {
|
||||
for (const libzcash::PaymentAddress& fa : filterAddresses) {
|
||||
const libzcash::SaplingPaymentAddress* sa = boost::get<libzcash::SaplingPaymentAddress>(&fa);
|
||||
if (sa == nullptr) {
|
||||
continue;
|
||||
}
|
||||
libzcash::SaplingIncomingViewingKey rivk;
|
||||
if (!GetSaplingIncomingViewingKey(*sa, rivk) || !IsCanonicalSaplingIvk(rivk)) {
|
||||
usePrefilter = false;
|
||||
break;
|
||||
}
|
||||
auto derived = rivk.address(sa->d);
|
||||
if (!derived || !(derived.get() == *sa)) {
|
||||
usePrefilter = false;
|
||||
break;
|
||||
}
|
||||
prefilterIvks.insert(rivk);
|
||||
}
|
||||
}
|
||||
|
||||
// Most wallet txs (coinbase, sends from other wallets) carry no notes of ours.
|
||||
for (const auto & p : mapWallet) {
|
||||
// A reference, not a copy: copying a CWalletTx deep-copies every note's witness list.
|
||||
// Nothing in this loop mutates mapWallet or any CWalletTx, and cs_main + cs_wallet are
|
||||
// held for the whole function, so the reference stays valid.
|
||||
const CWalletTx& wtx = p.second;
|
||||
|
||||
// A tx without Sapling note data can never yield an entry (the note loop below runs
|
||||
// zero times), so skip it before the finality, maturity and depth work.
|
||||
if (wtx.mapSaplingNoteData.empty())
|
||||
continue;
|
||||
|
||||
@@ -5942,97 +5726,84 @@ void CWallet::GetFilteredNotes(
|
||||
if (!CheckFinalTx(wtx) || wtx.GetBlocksToMaturity() > 0)
|
||||
continue;
|
||||
|
||||
// cs_main is held, so the depth can't change during the walk.
|
||||
const int nDepth = wtx.GetDepthInMainChain();
|
||||
if (minDepth > 1) {
|
||||
int nHeight = HeightFromDepth(nDepth);
|
||||
int nHeight = tx_height(wtx.GetHash());
|
||||
if ( nHeight == 0 ) {
|
||||
continue;
|
||||
}
|
||||
int nDepth = wtx.GetDepthInMainChain();
|
||||
int dpowconfs = hush_dpowconfs(nHeight,nDepth);
|
||||
if ( dpowconfs < minDepth || dpowconfs > maxDepth) {
|
||||
continue;
|
||||
}
|
||||
} else {
|
||||
if ( nDepth < minDepth || nDepth > maxDepth) {
|
||||
if ( wtx.GetDepthInMainChain() < minDepth ||
|
||||
wtx.GetDepthInMainChain() > maxDepth) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
nTxs++;
|
||||
|
||||
for (auto & pair : wtx.mapSaplingNoteData) {
|
||||
for (const auto & pair : wtx.mapSaplingNoteData) {
|
||||
const SaplingOutPoint& op = pair.first;
|
||||
SaplingNoteData& nd = pair.second;
|
||||
nNotes++;
|
||||
const SaplingNoteData& nd = pair.second;
|
||||
|
||||
// The cheap filters run before any decryption, which costs ~0.5ms of Jubjub arithmetic
|
||||
// per note. Spent notes stay in the wallet (-deletetx is off by default).
|
||||
if (ignoreSpent && nd.nullifier && IsSaplingSpent(*nd.nullifier)) {
|
||||
nSpent++;
|
||||
// Address pre-filter (see prefilterIvks above): a canonical ivk that is not one of
|
||||
// the verified filter ivks cannot generate any filter address, so the
|
||||
// filterAddresses check after decryption would reject this note anyway.
|
||||
if (usePrefilter && IsCanonicalSaplingIvk(nd.ivk) && prefilterIvks.count(nd.ivk) == 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Spent check before trial decryption: it reads only nd.nullifier and wallet
|
||||
// state (not the plaintext or the address), so spent notes are never decrypted.
|
||||
if (ignoreSpent && nd.nullifier && IsSaplingSpent(*nd.nullifier)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
uint256 pk_d;
|
||||
auto maybe_pt = SaplingNotePlaintext::decrypt_with_pk_d(
|
||||
wtx.vShieldedOutput[op.n].encCiphertext,
|
||||
nd.ivk,
|
||||
wtx.vShieldedOutput[op.n].ephemeralKey,
|
||||
wtx.vShieldedOutput[op.n].cm,
|
||||
pk_d);
|
||||
assert(static_cast<bool>(maybe_pt));
|
||||
auto notePt = maybe_pt.get();
|
||||
|
||||
// Same value as nd.ivk.address(notePt.d).get(): decryption succeeded, so
|
||||
// librustzcash_ivk_to_pkd(nd.ivk, d) returned true (d is a valid diversifier, which
|
||||
// is all check_diversifier tests) and wrote pk_d, the value address() recomputes.
|
||||
SaplingPaymentAddress pa(notePt.d, pk_d);
|
||||
|
||||
// skip notes which belong to a different payment address in the wallet
|
||||
if (!(filterAddresses.empty() || filterAddresses.count(pa))) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// skip notes which cannot be spent
|
||||
if (requireSpendingKey) {
|
||||
libzcash::SaplingIncomingViewingKey ivk;
|
||||
libzcash::SaplingFullViewingKey fvk;
|
||||
if (!(GetSaplingIncomingViewingKey(pa, ivk) &&
|
||||
GetSaplingFullViewingKey(ivk, fvk) &&
|
||||
HaveSaplingSpendingKey(fvk))) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
// skip locked notes
|
||||
if (ignoreLocked && IsLockedNote(op)) {
|
||||
LogPrint("zrpcunsafe", "%s: skipping locked note %s\n", __func__, op.hash.ToString().substr(0,10).c_str());
|
||||
continue;
|
||||
}
|
||||
if (ignoreLocked && IsLockedNote(op)) {
|
||||
LogPrintf("%s: skipping locked note %s\n", __func__, op.hash.ToString().substr(0,10).c_str());
|
||||
continue;
|
||||
}
|
||||
|
||||
assert(op.n < wtx.vShieldedOutput.size());
|
||||
const OutputDescription* output = &wtx.vShieldedOutput[op.n];
|
||||
candidates.push_back(Candidate { &op, &nd, output, nDepth });
|
||||
if (!nd.cachedPlaintext)
|
||||
uncached.push_back(SaplingNoteToDecrypt { &nd, output });
|
||||
// Same value as notePt.note(nd.ivk).get(), which is
|
||||
// SaplingNote(d, nd.ivk.address(d)->pk_d, value, rcm): pa is nd.ivk.address(d).
|
||||
libzcash::SaplingNote note(notePt.d, pa.pk_d, notePt.value(), notePt.rcm);
|
||||
saplingEntries.push_back(SaplingNoteEntry {
|
||||
op, pa, note, notePt.memo(), wtx.GetDepthInMainChain() });
|
||||
}
|
||||
}
|
||||
|
||||
// Decrypt what the cache lacks: normally nothing, since FindMySaplingNotes fills new notes and the
|
||||
// startup warm-up fills the ones loaded from wallet.dat.
|
||||
const unsigned int nFilled = uncached.empty() ? 0 : FillSaplingNoteCache(uncached);
|
||||
|
||||
// Pass 2: the filters that need the note's address.
|
||||
if (filterAddresses.empty())
|
||||
saplingEntries.reserve(saplingEntries.size() + candidates.size());
|
||||
for (const Candidate& c : candidates) {
|
||||
// A wallet note that doesn't decrypt under its own ivk is corrupt; as before, that's fatal here.
|
||||
assert(static_cast<bool>(c.nd->cachedPlaintext));
|
||||
if (fCheckNoteCache)
|
||||
CheckSaplingNoteCache(*c.op, *c.nd, *c.output);
|
||||
const SaplingNoteCache& plaintext = *c.nd->cachedPlaintext;
|
||||
const libzcash::SaplingPaymentAddress pa(plaintext.note.d, plaintext.note.pk_d);
|
||||
|
||||
// skip notes which belong to a different payment address in the wallet
|
||||
if (!(filterAddresses.empty() || filterAddresses.count(pa))) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// skip notes which cannot be spent
|
||||
if (requireSpendingKey) {
|
||||
libzcash::SaplingIncomingViewingKey ivk;
|
||||
libzcash::SaplingFullViewingKey fvk;
|
||||
if (!(GetSaplingIncomingViewingKey(pa, ivk) &&
|
||||
GetSaplingFullViewingKey(ivk, fvk) &&
|
||||
HaveSaplingSpendingKey(fvk))) {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
saplingEntries.push_back(SaplingNoteEntry {
|
||||
*c.op, pa, plaintext.note, plaintext.memo, c.nDepth });
|
||||
}
|
||||
|
||||
// A slow walk holds cs_main long enough to stall block connection, so it is always logged; a
|
||||
// fast one only with -debug=bench (the balance RPCs run it on every GUI poll). The wallet-size
|
||||
// counts go to the default log only with -debug=bench too.
|
||||
const int64_t nWalkMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - walkStart).count();
|
||||
if (LogAcceptCategory("bench")) {
|
||||
LogPrintf("%s: minDepth=%d ignoreSpent=%d requireSpendingKey=%d ignoreLocked=%d txs=%u/%u notes=%u spent=%u decrypted=%u entries=%u walk=%dms\n",
|
||||
__func__, minDepth, ignoreSpent, requireSpendingKey, ignoreLocked, nTxs, (unsigned int)mapWallet.size(),
|
||||
nNotes, nSpent, nFilled, (unsigned int)(saplingEntries.size() - nEntriesBefore), nWalkMs);
|
||||
} else if (nWalkMs >= 500) {
|
||||
LogPrintf("%s: minDepth=%d ignoreSpent=%d requireSpendingKey=%d ignoreLocked=%d walk=%dms\n",
|
||||
__func__, minDepth, ignoreSpent, requireSpendingKey, ignoreLocked, nWalkMs);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
+5
-56
@@ -62,8 +62,6 @@ extern bool bSpendZeroConfChange;
|
||||
extern bool fSendFreeTransactions;
|
||||
extern bool fPayAtLeastCustomFee;
|
||||
extern bool fTxDeleteEnabled;
|
||||
//! -checknotecache: re-decrypt cached Sapling notes in GetFilteredNotes and repair any difference
|
||||
extern bool fCheckNoteCache;
|
||||
extern bool fTxConflictDeleteEnabled;
|
||||
extern int fDeleteInterval;
|
||||
extern unsigned int fDeleteTransactionsAfterNBlocks;
|
||||
@@ -303,14 +301,6 @@ class SproutNoteData
|
||||
};
|
||||
|
||||
|
||||
/** A Sapling note's decrypted plaintext as the wallet uses it: the note (value, rcm, and the address it
|
||||
* was sent to as (d, pk_d)) and its memo. */
|
||||
struct SaplingNoteCache
|
||||
{
|
||||
libzcash::SaplingNote note;
|
||||
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
|
||||
};
|
||||
|
||||
class SaplingNoteData
|
||||
{
|
||||
public:
|
||||
@@ -338,15 +328,6 @@ public:
|
||||
// read true and short-circuit the witness self-heal in VerifyAndSetInitialWitness.
|
||||
bool witnessRootValidated = false;
|
||||
|
||||
//In Memory Only
|
||||
// The note's decrypted plaintext, so the balance and note-listing calls don't redo the Sapling
|
||||
// decryption (several Jubjub scalar multiplications) for every note on every call. It is a pure
|
||||
// function of (txid, output index, ivk), so nothing invalidates it. Never serialized, and left out
|
||||
// of operator== and SaplingNoteDataIdentical on purpose: a filled and an unfilled entry persist the
|
||||
// same state, so filling one must never make a tx look changed and get rewritten. Filled by
|
||||
// FindMySaplingNotes when the note is found, otherwise by FillSaplingNoteCache.
|
||||
boost::optional<SaplingNoteCache> cachedPlaintext;
|
||||
|
||||
ADD_SERIALIZE_METHODS;
|
||||
|
||||
template <typename Stream, typename Operation>
|
||||
@@ -385,11 +366,6 @@ struct SaplingNoteEntry
|
||||
int confirmations;
|
||||
};
|
||||
|
||||
/** Height of the active-chain block holding a wallet tx that is nDepth deep (as GetDepthInMainChain
|
||||
* returns it, and SaplingNoteEntry::confirmations holds it), or 0 when the tx is unconfirmed or not
|
||||
* in the active chain. Caller holds cs_main. */
|
||||
int HeightFromDepth(int nDepth);
|
||||
|
||||
/** A transaction with a merkle branch linking it to the block chain. */
|
||||
class CMerkleTx : public CTransaction
|
||||
{
|
||||
@@ -919,7 +895,7 @@ public:
|
||||
|
||||
protected:
|
||||
|
||||
int SaplingWitnessMinimumHeight(const boost::optional<uint256>& nullifier, int nWitnessHeight, int nMinimumHeight);
|
||||
int SaplingWitnessMinimumHeight(const uint256& nullifier, int nWitnessHeight, int nMinimumHeight);
|
||||
|
||||
/**
|
||||
* pindex is the new tip being connected.
|
||||
@@ -989,12 +965,8 @@ private:
|
||||
template <class T>
|
||||
void SyncMetaData(std::pair<typename TxSpendMap<T>::iterator, typename TxSpendMap<T>::iterator>);
|
||||
|
||||
// Last block ThreadNotifyWallets delivered to ChainTip, i.e. the block the wallet's state
|
||||
// reflects (guarded by cs_wallet). SetBestChain stamps this instead of the chain tip.
|
||||
const CBlockIndex* pindexWalletTip = nullptr;
|
||||
|
||||
protected:
|
||||
bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock);
|
||||
bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx);
|
||||
void MarkAffectedTransactionsDirty(const CTransaction& tx);
|
||||
|
||||
/* the hd chain data model (chain counters) */
|
||||
@@ -1238,10 +1210,12 @@ public:
|
||||
|
||||
/**
|
||||
* Get the wallet's activity log
|
||||
* @param fReadOnlyDb open wallet.dat read-only and without flush-on-close (pure readers such as
|
||||
* listtransactions: no txn_checkpoint per call); false = read-write, flush-on-close
|
||||
* @return multimap of ordered transactions and accounting entries
|
||||
* @warning Returned pointers are *only* valid within the scope of passed acentries
|
||||
*/
|
||||
TxItems OrderedTxItems(std::list<CAccountingEntry>& acentries, std::string strAccount = "");
|
||||
TxItems OrderedTxItems(std::list<CAccountingEntry>& acentries, std::string strAccount = "", bool fReadOnlyDb = false);
|
||||
|
||||
void MarkDirty();
|
||||
bool UpdateNullifierNoteMap();
|
||||
@@ -1322,8 +1296,6 @@ public:
|
||||
bool CommitAutomatedTx(const CTransaction& tx);
|
||||
/** Saves witness caches and best block locator to disk. Overrides CValidationInterface. */
|
||||
void SetBestChain(const CBlockLocator& loc);
|
||||
// Seed the block the wallet's state reflects, before ThreadNotifyWallets starts delivering.
|
||||
void SetWalletTip(const CBlockIndex* pindex);
|
||||
/** As SetBestChain, but for use INSIDE a long scan: opens the wallet DB with fFlushOnClose=false
|
||||
* so the call does not trigger a full BDB txn_checkpoint over the whole cache (see the comment
|
||||
* at CWallet::SetBestChain), and reports whether the write actually committed. */
|
||||
@@ -1488,29 +1460,6 @@ public:
|
||||
bool LoadCryptedMnemonicEntropy(const uint256& entropyFp, const std::vector<unsigned char>& vchCryptedSecret);
|
||||
|
||||
/* Find notes filtered by payment address, min depth, ability to spend */
|
||||
/** A wallet note whose decrypted plaintext is not cached yet, with the output it decrypts. */
|
||||
struct SaplingNoteToDecrypt {
|
||||
SaplingNoteData* nd;
|
||||
const OutputDescription* output;
|
||||
};
|
||||
/** Decrypts each note into nd->cachedPlaintext and returns how many it filled. A note that doesn't
|
||||
* decrypt under its ivk is left unfilled. Caller holds cs_wallet. */
|
||||
unsigned int FillSaplingNoteCache(const std::vector<SaplingNoteToDecrypt>& notes);
|
||||
/** Notes FillSaplingNoteCache has decrypted since startup (FindMySaplingNotes' fills not counted), and
|
||||
* the threads its last call used. Guarded by cs_wallet. */
|
||||
uint64_t nSaplingNoteCacheFills = 0;
|
||||
int nSaplingNoteCacheFillThreads = 0;
|
||||
|
||||
struct SaplingNoteCacheWarmup {
|
||||
unsigned int notes = 0; //!< notes in the wallet
|
||||
unsigned int filled = 0; //!< decrypted by this call
|
||||
unsigned int noSuchOutput = 0; //!< naming an output their tx doesn't have
|
||||
unsigned int undecryptable = 0; //!< not decrypting under their own ivk
|
||||
};
|
||||
/** Decrypts every wallet note whose plaintext isn't cached yet (all of them after wallet.dat is loaded).
|
||||
* Never asserts: a note that can't be decrypted is counted, logged and left unfilled. Run at startup. */
|
||||
SaplingNoteCacheWarmup WarmSaplingNoteCache();
|
||||
|
||||
void GetFilteredNotes(std::vector<SaplingNoteEntry>& saplingEntries,
|
||||
std::string address,
|
||||
int minDepth=1,
|
||||
|
||||
+14
-35
@@ -115,16 +115,23 @@ boost::optional<SaplingOutgoingPlaintext> SaplingOutgoingPlaintext::decrypt(
|
||||
}
|
||||
}
|
||||
|
||||
namespace {
|
||||
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
|
||||
const SaplingEncCiphertext &ciphertext,
|
||||
const uint256 &ivk,
|
||||
const uint256 &epk,
|
||||
const uint256 &cmu
|
||||
)
|
||||
{
|
||||
uint256 pk_d;
|
||||
return decrypt_with_pk_d(ciphertext, ivk, epk, cmu, pk_d);
|
||||
}
|
||||
|
||||
// SaplingNotePlaintext::decrypt with an incoming viewing key, also handing back the pk_d = [ivk]g_d
|
||||
// it derives to check cmu.
|
||||
boost::optional<SaplingNotePlaintext> DecryptWithIvk(
|
||||
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt_with_pk_d(
|
||||
const SaplingEncCiphertext &ciphertext,
|
||||
const uint256 &ivk,
|
||||
const uint256 &epk,
|
||||
const uint256 &cmu,
|
||||
uint256 &pk_d
|
||||
uint256 &pk_d_out
|
||||
)
|
||||
{
|
||||
auto pt = AttemptSaplingEncDecryption(ciphertext, ivk, epk);
|
||||
@@ -145,6 +152,7 @@ boost::optional<SaplingNotePlaintext> DecryptWithIvk(
|
||||
return boost::none;
|
||||
}
|
||||
|
||||
uint256 pk_d;
|
||||
if (!librustzcash_ivk_to_pkd(ivk.begin(), ret.d.data(), pk_d.begin())) {
|
||||
return boost::none;
|
||||
}
|
||||
@@ -165,39 +173,10 @@ boost::optional<SaplingNotePlaintext> DecryptWithIvk(
|
||||
return boost::none;
|
||||
}
|
||||
|
||||
pk_d_out = pk_d;
|
||||
return ret;
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
|
||||
const SaplingEncCiphertext &ciphertext,
|
||||
const uint256 &ivk,
|
||||
const uint256 &epk,
|
||||
const uint256 &cmu
|
||||
)
|
||||
{
|
||||
uint256 pk_d;
|
||||
return DecryptWithIvk(ciphertext, ivk, epk, cmu, pk_d);
|
||||
}
|
||||
|
||||
boost::optional<std::pair<SaplingNotePlaintext, SaplingNote>> SaplingNotePlaintext::decrypt_with_note(
|
||||
const SaplingEncCiphertext &ciphertext,
|
||||
const uint256 &ivk,
|
||||
const uint256 &epk,
|
||||
const uint256 &cmu
|
||||
)
|
||||
{
|
||||
uint256 pk_d;
|
||||
auto pt = DecryptWithIvk(ciphertext, ivk, epk, cmu, pk_d);
|
||||
if (!pt) {
|
||||
return boost::none;
|
||||
}
|
||||
// ivk_to_pkd succeeding implies d is a valid diversifier, which is all note(ivk) checks on top.
|
||||
SaplingNote note(pt->d, pk_d, pt->value(), pt->rcm);
|
||||
return std::make_pair(*pt, note);
|
||||
}
|
||||
|
||||
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
|
||||
const SaplingEncCiphertext &ciphertext,
|
||||
const uint256 &epk,
|
||||
|
||||
+11
-11
@@ -77,6 +77,17 @@ public:
|
||||
const uint256 &cmu
|
||||
);
|
||||
|
||||
// Same as decrypt(ciphertext, ivk, epk, cmu). On success it also stores in pk_d_out the
|
||||
// pk_d = [ivk] g_d that decryption computed, which equals ivk.address(d)->pk_d, so the
|
||||
// caller does not have to recompute it. pk_d_out is not written on failure.
|
||||
static boost::optional<SaplingNotePlaintext> decrypt_with_pk_d(
|
||||
const SaplingEncCiphertext &ciphertext,
|
||||
const uint256 &ivk,
|
||||
const uint256 &epk,
|
||||
const uint256 &cmu,
|
||||
uint256 &pk_d_out
|
||||
);
|
||||
|
||||
static boost::optional<SaplingNotePlaintext> decrypt(
|
||||
const SaplingEncCiphertext &ciphertext,
|
||||
const uint256 &epk,
|
||||
@@ -85,17 +96,6 @@ public:
|
||||
const uint256 &cmu
|
||||
);
|
||||
|
||||
// decrypt(ciphertext, ivk, epk, cmu) that also returns the note. Building the note needs
|
||||
// pk_d = [ivk]g_d, which the decryption derives anyway to check cmu; note(ivk) would derive it
|
||||
// again (another Jubjub scalar multiplication). A distinct name rather than an overload, so it
|
||||
// can never capture a call meant for the (epk, esk, pk_d, cmu) one.
|
||||
static boost::optional<std::pair<SaplingNotePlaintext, SaplingNote>> decrypt_with_note(
|
||||
const SaplingEncCiphertext &ciphertext,
|
||||
const uint256 &ivk,
|
||||
const uint256 &epk,
|
||||
const uint256 &cmu
|
||||
);
|
||||
|
||||
boost::optional<SaplingNote> note(const SaplingIncomingViewingKey& ivk) const;
|
||||
|
||||
virtual ~SaplingNotePlaintext() {}
|
||||
|
||||
Reference in New Issue
Block a user