DragonX/dragonx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Compare commits

merge into: DragonX/dragonx:perf/wallet-scan-rpc-costs
Branches Tags
DragonX/dragonx:master DragonX/dragonx:dev DragonX/dragonx:perf/wallet-scan-rpc-costs DragonX/dragonx:audit-fixes-20260901 DragonX/dragonx:autoshield-safety DragonX/dragonx:feature/autoshield-coinbase
DragonX/dragonx:v1.3.0 DragonX/dragonx:v1.2.0 DragonX/dragonx:v1.1.0 DragonX/dragonx:v1.0.3 DragonX/dragonx:v1.0.2 DragonX/dragonx:v1.0.1 DragonX/dragonx:v1.0.0
..
pull from: DragonX/dragonx:dev
Branches Tags
DragonX/dragonx:dev DragonX/dragonx:perf/wallet-scan-rpc-costs DragonX/dragonx:master DragonX/dragonx:audit-fixes-20260901 DragonX/dragonx:autoshield-safety DragonX/dragonx:feature/autoshield-coinbase
DragonX/dragonx:v1.3.0 DragonX/dragonx:v1.2.0 DragonX/dragonx:v1.1.0 DragonX/dragonx:v1.0.3 DragonX/dragonx:v1.0.2 DragonX/dragonx:v1.0.1 DragonX/dragonx:v1.0.0
25 Commits
perf/wallet-scan-rpc-costs .. dev
Author SHA1 Message Date
DanSandClaude Opus 5.5 3032eb7868 wallet/rpc: gather Sapling keys once per history RPC, not per output per address 
zsWalletTxJSON (z_listsentbyaddress, z_listreceivedaddress, getalldata)
runs once per wallet tx under LOCK2(cs_main, cs_wallet). For every
shielded output it copied the wallet's address set, and for each address
fetched the spending key, re-derived its full viewing key (an
elliptic-curve operation), and trial-decrypted the output; the sends side
did the same with the ovk. Every received or sent note in a tx with
spends walked the whole wallet in GetNullifiersForAddresses for the
change check. The spends side decrypted every spent note again and
deep-copied its SaplingNoteData (witnesses included). The GUI runs
z_listsentbyaddress over the whole wallet on every launch.

SaplingHistoryKeys, built once per RPC call, holds each address with a
usable spending key (in GetSaplingPaymentAddresses order, with its ivk
and ovk as the spending key gives them, and its ivk as
GetSaplingIncomingViewingKey gives it). It also gives the change check
as a lookup into the wallet's (ivk, nullifier) pairs, collected on first
use. The helpers keep their exact structure and quirks. Each output's
JSON object and change flag still accumulate across the addresses that
decrypt it. Diversified addresses sharing a key still produce one entry
each. An unknown address still checks change under a zero ivk. A locked
wallet still reports no shielded received or sent entries. Only the
repeated work is gone:

- each output is decrypted at most once per distinct key. Once it is
  known which ivk decrypts an output, no other ivk is tried: the cached
  plaintext, or a decryption under the ivk the wallet's note data
  records. Two canonical ivks can't both pass the note commitment check,
  which pins pk_d = [ivk]g_d;
- outCiphertexts are decrypted once per distinct ovk; the t->z shielding
  ovk is derived from the seed once per tx;
- spends use the cached plaintext of the spent note.

Two deliberate behaviour changes. The spends helper looked notes up with
operator[]: it inserted a null-outpoint entry into
mapSaplingNullifiersToNotes for every other wallet's nullifier it met.
After getalldata or z_listsentbyaddress had run, z_viewtransaction of
any tx spending someone else's note (every incoming z->z payment) failed
with map::at. And with an inconsistent nullifier map, it inserted an
empty note into a wallet tx. The helpers now only read the wallet. Also,
if the walletpassphrase timer relocks the wallet during the call, the
whole call reports the keys as of its start; the old per-output lookups
switched over partway.

test_wallet_history.cpp holds 69bb035e7's helpers verbatim (renamed)
and a wallet exercising every branch:
- default and diversified addresses, a viewing-key-only address, and a
  key imported after its note arrived;
- z->z sends with change, and with text, empty and binary memos;
- a send nobody's ovk reads, and a t->z shielding send;
- our spends, someone else's, and a mixed tx.
Every tx x every return type x nine address filters gives byte-identical
JSON: with the plaintext cache warm, cold, and on an encrypted wallet
both locked and unlocked. Mutation-checked: dropping the ivk check from
the uniqueness shortcut fails the test. DoesNotModifyTheWallet pins the
read-only change (and shows that the old helpers did insert the foreign
nullifier).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 7c3563cd5a wallet/rpc: order history RPC txs by pointer, not by deep copy 
getalldata, z_listreceivedaddress and z_listsentbyaddress build a
map<int64_t, CWalletTx> of the whole wallet, keyed by nOrderPos, before
any filter runs. Every insert deep-copies the CWalletTx, including each
note's witness list (~100KB per note on a mainnet wallet). That is
gigabytes of allocation per call on a large wallet, under
LOCK2(cs_main, cs_wallet). The GUI's history scan runs
z_listsentbyaddress on every launch and about once per new tip.

The map now holds const CWalletTx* into mapWallet. It is still a
std::map, so the first tx inserted for an nOrderPos wins, exactly as
before. mapWallet can't change while the RPC holds cs_wallet, so the
pointers stay valid. Output is unchanged for a consistent wallet. One
corner case differs: with a nullifier map entry pointing at a note
its tx lacks, the spends helper's operator[] inserts an empty note into
that tx, and a later iteration now sees it where the old copies didn't.
The next commit removes that insertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 190023f85f wallet: note-cache review follow-ups 
- UpdatedNoteData now keeps the incoming plaintext when it keeps the
  wallet's own entries. When it returns "unchanged" (equal note data,
  or SaplingNoteDataIdentical), any entry lacking a plaintext takes the
  incoming one, for the same outpoint and ivk only. The startup rescan
  replays loaded txs with FindMySaplingNotes' freshly decrypted data;
  that plaintext was thrown away, so the warm-up decrypted those notes a
  second time. The cache is outside every comparison, so the function
  still returns false and nothing is rewritten.
- -checknotecache is read once at init into fCheckNoteCache, instead of
  a mapArgs lookup on every GetFilteredNotes call. Its help text and
  comment now say what it checks: every note the walk considers, before
  the address filter, so a corrupt cached address can't drop a note
  unchecked.
- GetFilteredNotes pre-sizes its result only when there's no address
  filter, so a one-address call no longer reserves room for the whole
  wallet.
- WarmSaplingNoteCache returns its counts (notes, filled, no such
  output, undecryptable), and FillSaplingNoteCache records the threads
  it used, so tests can tell the cases apart.
- WarmSaplingNoteCache's comment no longer claims nothing can wait on
  cs_wallet while it runs: the RPCs allowed during warmup can.

Tests:
- ReplayWithPlaintextIsNoUpdate pins the rule that the plaintext cache
  stays out of the "did this tx change?" decision. A rescan replays a
  witnessed note loaded without plaintext, using FindMySaplingNotes'
  data that carries one. That must be no update: no rewrite, and the
  witness and nullifier are kept. The stored entry then gains the
  plaintext. Checked by mutation: with cache presence added to
  SaplingNoteDataIdentical, this test fails and every other test still
  passes.
- ParallelFillMatchesSerial uses 460 notes and asserts the thread count.
  So the 7-thread case really runs 7 threads, with uneven shards
  (460 % 7 != 0). Before, it was capped at 4 and repeated the 4-thread
  case.
- ColdParallelFillMatchesLegacy: a cold GetFilteredNotes over ~270
  candidates fills on 4 threads and matches the 69bb035e7 loop entry
  for entry.
- WarmUpSkipsBadNotes asserts the warm-up's separate counts (1 no such
  output, 2 undecryptable), so removing the out-of-range guard is now
  caught.
- CachedPlaintextMatchesFreshDecrypt checks exact counts.
  CheckNoteCacheRepairsADifference also tampers with a memo, and looks
  entries up by outpoint. The meaningless GetHash check is dropped.
- A ScopedArg / ScopedCheckNoteCache RAII helper restores -args and
  fCheckNoteCache even when an assertion returns early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 23877e1038 wallet: -checknotecache, a diagnostic for the note plaintext cache 
With -checknotecache=1 (debug help), GetFilteredNotes decrypts every
note it returns again. If the cached plaintext differs from the fresh
decryption, it logs the note and replaces the cache entry with the
fresh value. It costs the full decryption the cache exists to avoid, so
it is off by default; it is there for validating the cache on a real
wallet, or for diagnosing a suspected cache problem in the field.

CheckNoteCacheRepairsADifference: by default a tampered cache entry is
returned as is (the cache is trusted); with the option it is repaired
and the fresh value is returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 43635b7fd7 wallet: decrypt big note-cache batches on worker threads 
The startup warm-up, or a lazy fill after -notecachewarmup=0, decrypts
the whole wallet: ~0.5ms per note, ~6s serial for 12k notes. Each
decryption is independent: it reads its own output and writes only its
own note's cachedPlaintext. So FillSaplingNoteCache now splits batches
of 128 or more notes round-robin across -witnessbuildthreads (default:
hardware_concurrency) workers, at least 64 notes per thread. It follows
BuildWitnessCache's Phase 2 pattern:

- the caller keeps cs_wallet held throughout;
- workers take no lock and assert none; under DEBUG_LOCKORDER a fresh
  thread has no lock stack;
- per-shard fill counters are summed after join;
- nothing escapes a worker: a throwing decryption leaves that note
  unfilled, like one that doesn't decrypt;
- if the OS refuses a thread, the remaining shards run inline.

librustzcash is already called concurrently by BuildWitnessCache's
workers, and libsodium is thread-safe after sodium_init.

ParallelFillMatchesSerial: 300 notes across two keys and diversified
addresses decrypt to exactly the same cache with 1, 4 and 7 threads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 83a14a7bfb wallet/init: decrypt every wallet note into the plaintext cache at startup 
The plaintext cache is in memory only, so a restart empties it. Without
a warm-up, the first balance call after a restart decrypts the whole
wallet under LOCK2(cs_main, cs_wallet): ~6s for a 12k-note wallet. That
is the observed failure: restart, the GUI connects, and the node looks
stuck.

CWallet::WarmSaplingNoteCache decrypts every note whose plaintext isn't
cached, spent ones included, because z_listreceivedbyaddress
(ignoreSpent=false) is called once per address by the GUI's history
scan.

init runs it at the end of the wallet block, just before
SetBroadcastTransactions. At that point RPC is still in warmup (it
shows "Decrypting wallet notes...") and the wallet notification
thread, whose startup witness rebuild can hold cs_wallet for a long
time, has not started yet. It takes cs_wallet only. -notecachewarmup=0
(debug help) skips it, which leaves the lazy fill in GetFilteredNotes
to do the work.

Unlike GetFilteredNotes, the warm-up never asserts. A note that doesn't
decrypt under its ivk, or names an output its tx lacks, is counted in
the summary line and listed under -debug=zrpcunsafe. A corrupt note
therefore can't stop the node from starting, or from serving the
key-export RPCs allowed during warmup; GetFilteredNotes still asserts
on it, as before.

Tests:
- WarmUpFillsEveryNote: spent, locked, unconfirmed and stale-block
  notes included; walks afterwards decrypt nothing.
- WarmUpSkipsBadNotes: undecryptable and out-of-range notes are skipped
  without aborting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 b1e3c5230a wallet: cache each Sapling note's decrypted plaintext in memory 
GetFilteredNotes, the walk behind every balance and note-listing RPC,
sweep, consolidation, z_sendmany and z_mergetoaddress, decrypted every
note it looked at on every call: ~0.5ms of Jubjub arithmetic per note,
under LOCK2(cs_main, cs_wallet). A note's plaintext is fixed by its
output (the txid commits to the ciphertext) and the ivk that decrypts
it, so it never needs to be decrypted twice.

SaplingNoteData gains an in-memory-only
boost::optional<SaplingNoteCache> {note, memo}; the address is the
note's (d, pk_d). It follows the witnessRootValidated precedent:
- never serialized, and left out of operator== and
  SaplingNoteDataIdentical, so filling it never makes a tx look changed
  and get rewritten (the replay rewrite storm 69bb035e7 removed);
- FindMySaplingNotes fills it from the decryption that found the note
  (both the spending-key and the viewing-key-only loop);
- UpdatedNoteData's replace branch keeps a plaintext the wallet already
  had, for the same outpoint and ivk only;
- GetFilteredNotes now runs in two passes: pass 1 applies the tx,
  spent and locked filters and collects the notes that pass; whatever
  lacks a plaintext (everything after a restart, until the startup
  warm-up in a later commit) is decrypted once via the new
  FillSaplingNoteCache; pass 2 applies the address and spending-key
  filters and emits entries in the same order as before.
Spentness, spendability, lock state and depth are still decided on every
call; only the plaintext is cached.

The A0 equivalence test passes unchanged (warm). New tests:
- ColdCacheMatchesLegacy: the same grid with the cache dropped before
  every call;
- CachedPlaintextMatchesFreshDecrypt: eager and lazy fills;
- WarmPassDoesNotDecrypt: a fill counter shows a warm walk decrypts
  nothing, and spent or locked notes only when the caller asks for
  them;
- CacheNeverSerialized: byte-identical SER_DISK/NETWORK/GETHASH output,
  empty after a load;
- UpdatedNoteDataAndTheCache: no rewrite on a cache-only difference,
  carried on replace, never across another ivk;
- SpendabilityNotCached;
- FindMySaplingNotesFillsCache.

Memory: ~616 bytes per note (the memo is 512), ~8MB for a 13k-note
wallet, next to ~100KB per note of cached witnesses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 7ddd03d86d zcash/wallet: decrypt a Sapling note's plaintext and note in one pass 
SaplingNotePlaintext::decrypt(ciphertext, ivk, epk, cmu) derives
pk_d = [ivk]g_d to check the note commitment, then throws it away. The
wallet then calls ivk.address(d), which derives pk_d again, and
plaintext.note(ivk), which derives it a third time. Each derivation is a
Jubjub scalar multiplication, ~115us, measured. Including the key
agreement and the commitment check, the whole sequence is ~775us per
note.

decrypt_with_note() runs the same decryption and checks and returns the
note built from that pk_d. ivk_to_pkd succeeding already implies a
valid diversifier, which is all ivk.address() checks on top. It gets its
own name rather than a decrypt() overload: an overload taking a fifth
argument by non-const reference would capture calls meant for the
(epk, esk, pk_d, cmu) variant. The cmu check stays.

GetFilteredNotes and FindMySaplingNotes' spending-key loop use it. That
saves two of the three derivations per note in every balance walk, and
one per found note during scans and rescans (trial decryption of
outputs that aren't ours still fails at the AEAD, as before).
FindMySaplingNotes also no longer copies each OutputDescription.

New tests: MatchesDecryptThenNote checks decrypt_with_note field for
field against decrypt() + note(ivk), over default and diversified
addresses, a wrong ivk and a wrong cmu. FindMySaplingNotes pins what
discovery returns: notes per key, diversified addresses to register,
foreign outputs ignored. Both pass before and after the switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 bfec5c74ae gtest: notecache review fixes 
- Take LOCK2(cs_main, cs_wallet) before adding keys in SetUp: both
  AddSaplingZKey and AddSaplingIncomingViewingKey assert cs_wallet is
  held, which aborts a DEBUG_LOCKORDER build.
- TIP_HEIGHT becomes a namespace-scope constexpr. As a static class
  member it was ODR-used by EXPECT_EQ without a definition, which fails
  to link as C++11 at -O0.
- TearDown restores the previous chainActive tip and erases only the
  mapBlockIndex entries the fixture inserted.
- Add a tx mined only in a stale side block. The legacy tx_height model
  returns that block's height, as a txindex can, and the tx must still
  be excluded at every minDepth.
- The encrypted variant asserts HaveSaplingSpendingKey directly, and a
  new test unlocks the keystore and checks GetSaplingSpendingKey (the
  other loop the crypter change touched).
- The legacy-loop comment now says its two LogPrintf lines are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 21586b353a wallet/rpc: monotonic timers and a quieter default slow-walk line 
Review follow-ups to the GetFilteredNotes / RPC timing logs:

- Time with std::chrono::steady_clock instead of GetTimeMicros(), which
  is the system clock. A wall-clock step otherwise writes false
  ">= 500ms" lines unconditionally; on a WSL2 host the clock was seen
  stepping +3.6s every ~32s.
- A long-poll getblocktemplate deliberately blocks (with cs_main
  released) until a new block arrives, so it is logged only under
  -debug=bench.
- Log the method as "RPC method=<name> took <n>ms". A bare " rescan "
  token followed by "<n>ms" matches ObsidianDragon's rescan-finished
  log heuristic.
- The always-on slow-walk line keeps just the filters and the walk
  time; the wallet-size counts (txs, notes, spent, entries) are logged
  only with -debug=bench, so a shared debug.log doesn't carry them.
- The -txindex=0 startup line names the version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 34b94e761a wallet: don't copy every encrypted Sapling key on each lookup 
CCryptoKeyStore::HaveSaplingSpendingKey and GetSaplingSpendingKey scan
mapCryptedSaplingSpendingKeys with `for (auto entry : ...)`. That copies
each entry (an extended full viewing key plus its encrypted-secret
vector) for every key visited, on every call. GetFilteredNotes'
requireSpendingKey check makes one such call per note, and the history
RPCs make one per output per address: ~22ns per key per call, measured
(11us per note on a 1024-key wallet). Iterate by const reference.

The equivalence test gains a locked, encrypted-keystore variant,
LockedWalletMatchesLegacy, so spendability coming from the encrypted
map is covered too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 94967efa7b wallet: take note heights from the wallet's depth, not tx_height() 
For minDepth > 1, GetFilteredNotes located each wallet tx's block with
tx_height(), which calls GetTransaction and re-reads the tx from disk
through the txindex. It did this once per wallet tx, transparent ones
included, while holding LOCK2. z_listunspent and z_listreceivedbyaddress
did it once more per returned note. Without -txindex, GetTransaction can
only find a tx that still has unspent transparent outputs. tx_height()
then returned 0 for every fully shielded tx, and the tx was skipped. So
on a -txindex=0 node, sweep and consolidation (minDepth 11) and any
minconf >= 2 call never saw those notes.

The wallet already has the depth, and for an active-chain tx
GetDepthInMainChain is chainActive.Height() - height + 1. The new
HeightFromDepth() inverts that (0 when unconfirmed or off the active
chain) and replaces tx_height() at all three sites. The height only
feeds the == 0 skip and hush_dpowconfs. Both sides hold cs_main for the
whole call, so the tip can't move between the walk and the RPC tail.

On a -txindex node the results are identical: the txindex block and
wtx.hashBlock are the same block. The extended equivalence test now
also covers minDepth 2/11/12/22. On -txindex=0 nodes, fully shielded
notes now show up for minDepth > 1. With -zsweep or -consolidation
enabled, those notes will now be swept and consolidated, so init logs
that once. MinDepthAboveOneUsesWalletHeight pins the new behaviour and
shows that the old path returned nothing without a txindex.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 bfc1334565 wallet: walk GetFilteredNotes by reference, cheap filters first 
GetFilteredNotes copied every CWalletTx by value before any filter ran,
then copied each SaplingNoteData by value again. Both copies deep-copy
the note's witness list (up to WITNESS_CACHE_SIZE witnesses, ~100KB per
note on a mainnet wallet): ~40us per note per pass, measured. Spent and
locked notes were also fully decrypted (~0.6ms of Jubjub arithmetic per
note) before the spent and locked checks threw them away. -deletetx is
off by default, so spent notes accumulate.

- iterate mapWallet and mapSaplingNoteData by const reference;
- skip txs with no notes of ours before CheckFinalTx;
- read GetDepthInMainChain once per tx instead of 2+N times (cs_main is
  held, so it can't change during the walk);
- run the spent and locked checks before the decryption;
- z_listreceivedbyaddress no longer copies the whole CWalletTx per
  entry to read its time, and z_sendmany's sort comparator takes its
  entries by reference.

The filters have no side effects, so the result set and its order are
unchanged; the A0 equivalence test passes unchanged. One deliberate
difference: a spent or locked note whose ciphertext no longer decrypts
under its ivk is now skipped instead of tripping the decrypt assert. The
new SpentAndLockedNotesAreNotDecrypted test pins this (it aborts on the
old ordering).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 b9cdce8e96 wallet/rpc: time GetFilteredNotes walks and slow RPCs 
GetFilteredNotes logged "GetFilteredNotes ignoreLocked=N" on every call.
The balance RPCs run it on every GUI poll, so that line was log spam on
a fast wallet, and on a slow one the gaps between its timestamps were
the only timing there was. Those gaps also count lock waits and the
previous RPC's own post-processing.

It now logs one summary per walk: the filters, txs that passed the
tx-level checks out of the wallet total, notes examined, notes skipped
as spent, entries returned, and the walk time. The summary is always
written when the walk takes >= 500ms (that long under cs_main stalls
block connection) and otherwise only with -debug=bench. The line has no
lock-wait figure: every caller already holds LOCK2(cs_main, cs_wallet)
and CCriticalSection is recursive, so the inner LOCK2 never waits.

Lock waits are measured where they happen instead. CRPCTable::execute
logs "RPC <method> took <n>ms" under the same >= 500ms / -debug=bench
rule. The per-locked-note line moves to -debug=zrpcunsafe, since it
logs txid prefixes.

No behaviour change otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 dc8ca4996a gtest: pin GetFilteredNotes behaviour before optimizing it 
Adds test_wallet_notecache.cpp: a wallet with Sapling notes at several
depths (spent by a mined tx, spent by an unmined tx, locked, viewing-key
only, never mined, plus txs carrying no wallet notes), mined into a fake
active chain. The fixture undoes everything it puts into the process-wide
chain state (chainActive, mapBlockIndex, pcoinsTip) in TearDown, since all
gtest suites share one process.

LegacyFilteredNotes is the 69bb035e7 loop verbatim, with tx_height()
passed in so the minDepth > 1 branch can be modelled on a -txindex node.
FilteredNotesMatchLegacy compares the real walk against it entry by entry
and in order (z_sendmany sorts the result with an unstable sort) over
minDepth {0,1} x maxDepth x ignoreSpent x requireSpendingKey x
ignoreLocked x address filters. FixtureExercisesEveryFilter checks that
each filter really removes something, so the comparison can't pass
vacuously.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 b911189ac6 wallet: don't witness viewing-key-only notes (fixes an abort on the next block) 
A note found through an incoming viewing key only (z_importviewingkey)
has no full viewing key behind it. UpdateSaplingNullifierNoteMapWithTx
skips such notes, so they never get a nullifier. VerifyAndSetInitialWitness
treated "no nullifier" as "witness needs reseeding" on every call: it
cleared the note's witness, re-read the block from disk, reseeded it,
and then dereferenced the nullifier that never came. dragonxd aborted
in the txnotify thread (boost::optional "is_initialized" assert) on the
first block confirming a note to such an address, and again after every
restart, before the block index was flushed.

Such a note can't be spent, so it needs no witness. Skip it there; if
its spending key is imported later, it has a full viewing key and the
existing no-nullifier rule seeds it then. SaplingWitnessMinimumHeight
now takes the optional nullifier. A note without one is never extended
by BuildWitnessCache (its pending-note gate requires one), so it must
not lower the rebuild's start height. Every call site that worked before
passes a set nullifier and behaves as before.

The deref came in with the 2020 witness rework (6c48397cf). The new
gtest ViewingKeyOnlyNoteIsNotWitnessed fails on the old code (the note
reaches the reseed path).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 7131b3673f qa: add fd-guard, the close() guard and regtest harness that found the double close 
shim/ is an LD_PRELOAD guard around close/fclose/dup2/dup3:
- before each close it resolves /proc/self/fd/N;
- it reports, with a backtrace, a non-owner close of wallet.dat,
  database/log.* or blocks/*.dat, and every close that fails with EBADF
  (the second half of a double close);
- FDGUARD_ABORT=1 aborts on the first hit;
- run_tests.sh self-tests it (51 checks).

repro/ runs a shimmed dragonxd on isolated regtest. Each run gets its
own unprivileged network namespace (loopback only). The loads are:
- rpc: a hammer with impatient/RST/half-open clients, work-queue
  bursts and long-polls;
- p2p: TLS peers churned by disconnectnode, ban and kill, plus a raw
  TCP/TLS misbehaver;
- tor: a fake control port;
- mine: mining with blocknotify and autoshield.
tools/tally.py groups the reports by closer. tools/scan_bdb.py finds BDB
pages written into block files.

With the wolfSSL CloseSocket macro in effect:
- p2pmin (300 s) gives one EBADF double close per disconnect;
- the full load (1200 s) gives 930 EBADF closes, 186 non-owner closes
  (37 of wallet/log/blk files) and a 632 KB db.log with BDB0060 PANIC;
- RPC-only and Tor-only loads give none.
After the fix, every count is 0 and db.log stays empty. The README has
the commands and the numbers.

Paths come from the environment (GOLDEN, BIN, RUNS, SHIM, ASMAP and the
Sapling params). Run output goes to $RUNS, outside the tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 21:23:01 -05:00
DanSandClaude Opus 5.5 70b35ac11a wallet/db: keep the BDB logs when the shutdown flush fails 
CDBEnv::Flush(true) removed database/ (log_archive DB_ARCH_REMOVE, then
remove_all) once mapFileUseCount was empty. It did that even when the
checkpoint or lsn_reset before it had failed, or the environment had
panicked (BDB0060). In those states wallet.dat can be missing committed
changes that only the logs hold:
- a page write failed (EBADF/ESPIPE through an fd closed behind BDB's
  back, the wolfSSL CloseSocket bug), so the page stayed dirty in the
  mpool cache and the checkpoint never completed;
- pages written after the last lsn_reset carry real LSNs that point
  into those logs.
Deleting the logs turned a wallet that the next start's DB_RECOVER would
repair into one that silently lost those changes. A std::terminate kept
the logs; the clean shutdown that ThreadNotifyWallets now does on a
wallet DB error deleted them.

Remember a failed txn_checkpoint or lsn_reset (in Flush and in
CheckpointLSN, which the periodic flush and backupwallet use). Before
removing anything, also check the environment for a panic. get_flags
reports it, and itself fails with DB_RUNRECOVERY once the environment
has panicked. In any of those cases keep database/ and log why. If the
kept logs are unusable, the next start already moves database/ aside
and retries (CWallet::Verify).

WalletDbEnv tests:
- a clean shutdown still removes the logs;
- a panicked environment (DB_PANIC_ENVIRONMENT) keeps them;
- a failed lsn_reset keeps them.
With the old db.cpp only the first passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 21:23:01 -05:00
DanSandClaude Opus 5.5 5b36d7ad24 validationinterface: shut down cleanly when a wallet block notification throws 
ThreadNotifyWallets' block disconnect and connect loops had no
try/catch (only its mempool loop does). A wallet database failure
inside SyncWithWallets or ChainTip escaped the thread, and TraceThread
logs and rethrows, so the node died in std::terminate with nothing
flushed. One example: CWalletDB::ListAccountCreditDebit's "error
scanning DB" after BDB lost its file to the wolfSSL CloseSocket bug.
That is how the pool's node died on 2026-10-12 and 2026-10-16.

Run each block's wallet notifications through NotifyWalletsOrShutdown.
On an exception it logs the error, requests a shutdown, shows the usual
fatal-error message and stops the notifier, leaving pindexLastTip at
the last block the wallet fully processed. Carrying on instead would
leave the wallet behind the chain, missing the block it failed on.
Thread interruption still propagates. The shutdown is requested before
the fatal-error message is shown.

FdDefense.WalletNotifyFailureShutsDownCleanly:
- a throwing notification requests a shutdown, a clean one doesn't;
- a non-std exception is handled the same way;
- boost::thread_interrupted still propagates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 21:23:01 -05:00
DanSandClaude Opus 5.5 9cb31326e3 main: open block and undo files read-only when they're opened for reading 
OpenDiskFile opened every block or undo file "rb+", even for its
fReadOnly callers, so every reader held a writable fd. That's what let
the wolfSSL CloseSocket bug corrupt blocks/*.dat. A P2P double close
took the wallet's BDB fd away; the number was then handed to a block
file opened for a getblock/getdata read; BDB's next wallet page write
landed in it, at offset pgno*4096. On production that destroyed blocks
2,728,132-2,728,161 in blk00011.dat and block 3,356,880 in blk00020.dat.

Every fReadOnly caller only reads through the handle:
- ReadBlockFromDisk, UndoReadFromDisk, both GetTransaction paths and
  hush_blockload read via CAutoFile >>;
- the reindex loop hands it to LoadExternalBlockFile, which reads via
  CBufferedFile;
- the existence check at startup only opens it;
- hush_prefetch only freads.
Open those "rb". A stray write through a reused fd number then fails
(EBADF) instead of silently corrupting a block file. Missing files
still fail to open as before; writers keep "rb+"/"wb+".

FdDefense.ReadOnlyBlockFileIsNotWritable: pwrite through a read-only
block file's fd fails with EBADF, while the writable open still writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 21:23:01 -05:00
DanSandClaude Opus 5.5 d91e069da3 net: stop wolfSSL's CloseSocket macro from turning P2P closes into double closes 
wolfssl/wolfio.h (included by <wolfssl/ssl.h> in net.h) does
`#ifndef CloseSocket / #define CloseSocket(s) close(s)`. Every TU that
includes net.h after that point compiles CloseSocket(x) to a bare
close(x). That includes all 15 calls in net.cpp, among them the ones
in CloseSocketDisconnect and ~CNode. util/sock.cpp's
CloseSocket(SOCKET&), the only thing that sets the handle to
INVALID_SOCKET, was never called from net.cpp. This has been the case
since OpenSSL was replaced by wolfSSL (Hush 9ff756590, 2020), so every
DragonX release is affected.

So CNode::hSocket kept its number after the socket was closed. Every
P2P disconnect closed that same number again, 2 to 4 times:
- CloseSocketDisconnect (net.cpp:572) runs once per disconnect trigger
  (ThreadSocketHandler, SocketSendData, the TLS socket handler,
  ThreadMessageHandler), and each run still saw a valid-looking hSocket;
- ~CNode (net.cpp:2819) then closed it once more.
Meanwhile the socket handler kept FD_SETting, send()ing and recv()ing on
the stale number. In between, other threads get that number from the
kernel's lowest-free-fd rule:
- BDB reopening wallet.dat or database/log.* (ThreadFlushWalletDB, ~11/s);
- fopen("rb+") of a block file for getdata/getblock;
- an HTTP RPC accept.
The stray close then closes it. BDB keeps doing I/O on the dead number:
'wallet.dat: write failed for page N' / 'Illegal seek' / 'BDB0060
PANIC', or silent wallet-page writes into a blocks/blk*.dat that got
the number next. Its failing reads later threw out of ThreadNotifyWallets
(std::terminate). A closed HTTP connection fd makes libevent re-arm
and later close whatever owns that number next ('Epoll ADD ... failed').

Reproduced on isolated regtest with an LD_PRELOAD close() guard:
- one disconnectnode gives exactly one EBADF double close (49
  disconnects, 50 EBADF; with strace and no shim, 29 disconnects gave
  30 'close(N) = -1 EBADF');
- P2P churn alone gives non-owner closes of block files and wallet.dat;
- RPC-only and Tor-only loads give none.
Disassembly of CloseSocketDisconnect and ~CNode shows 'call close@plt'
before this change and 'call CloseSocket(unsigned int&)' after.

Undefine the macro right after the wolfSSL include in net.h; wolfio.h
is include-guarded, so it can't come back. net.cpp gets an #error
tripwire in case it ever does. NetCloseSocket.ClosedNodeNeverClosesAReusedFdNumber
fails on the old code (a closed CNode closes the fd number's new owner)
and passes now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 21:23:00 -05:00
DanSandClaude Opus 5.5 69bb035e7e wallet/main: break the replay -> witness-rebuild -> kill loop on large wallets 
A v1.3.0 node with a ~13.6k-note wallet (-dbcache=4084) looked hung for hours:
every start replayed ~62k blocks from a chainstate 26 days stale, then ran a
47-60 min all-core witness rebuild holding LOCK2(cs_main, cs_wallet) (RPC answered
HTTP 500 "Work queue depth exceeded", peers timed out), the GUI killed it before
anything was flushed, and the next start did it all again. Even after escaping,
each new block cost ~30s under LOCK2, so the node fell behind a 36s chain.

D1  BuildWitnessCache: the post-rebuild latch computed witnesses.front().root()
    (~2.4ms) for every pending note on every block, only to feed a log line that
    is printed for bulk rebuilds alone. The anchor check now runs on the Phase-2
    workers and only for a bulk rebuild; the per-block path just sets the latch.
    Phase 2 also runs in parallel whenever the job is big (notes x commitments
    >= 2000), not only for a bulk range, so busy blocks no longer extend every
    note serially. If the OS refuses a worker thread, the remaining shards run
    inline instead of unwinding past joinable threads (std::terminate).

D2  FlushStateToDisk: the coins cache was only flushed on shutdown, gettxoutsetinfo
    or after 24h; on this chain it stays a few MiB, so the size triggers never
    fire and a node that never reached 24h uptime or a clean stop kept its
    chainstate (and block index) weeks behind. Flush at least hourly and every
    DATABASE_FLUSH_BLOCKS (5000) connected blocks -- the latter also inside one
    long ActivateBestChain -- and right after the startup ActivateBestChain when
    it moved the tip.

D3  UpdatedNoteData: a replayed or rescanned tx arrives with FindMySaplingNotes
    data (no nullifier). Witnesses and witnessHeight were carried over but the
    nullifier was dropped, so VerifyAndSetInitialWitness cleared and reseeded the
    witnesses on the next block -- the replay above rewrote 3,750 txs and forced
    the full rebuild. Carry the nullifier with the witnesses, unless the tx was
    re-mined in a different block (its notes moved; reseed as before). When the
    merged data is identical, don't rewrite the tx; just clear the in-memory
    witnessRootValidated so the next block re-checks the kept roots once (a
    rescan still finds and reseeds a witness that went bad, without rebuilding
    the healthy ones). The carry loop no longer deep-copies every note.

    SetBestChain: with hourly coin flushes the wallet locator must not run ahead
    of the async notifier, or a crash permanently skips the blocks it had not yet
    delivered. Stamp the block the wallet's state reflects (pindexWalletTip,
    seeded in init before ThreadNotifyWallets starts -- at the startup rescan's
    checkpoint if that rescan was interrupted -- and advanced contiguously by
    ChainTip), like zcashd's ChainTipAdded. ThreadNotifyWallets no longer hands
    the wallet an empty block after a fatal block-read failure.

Tests: new gtest test_wallet_notedata.cpp (5 tests); 23/23 gtests pass, and the
two core D3 tests fail with D3 reverted. Regtest A/B against master with a
1,008-note wallet: notify-thread CPU per block 1,780ms -> 3ms; worst LOCK2 RPC
stall 1.6-2.9s -> 10-15ms on ordinary blocks and 4.6-7.1s -> ~1.6s on a block
carrying a 40-output self-send; a z_importkey rescan went from 35 reseeds plus a
bulk rebuild (10.6s) to 0 reseeds, same balance; after kill -9 at 150 blocks a
build with a 100-block flush threshold restarted at 100, master at 0. Two
adversarial review rounds; every confirmed finding is addressed here.

Not covered: the hourly SetBestChain/flush paths were reviewed, not exercised
(GetTimeMicros ignores mocktime); no run yet against a copy of a real large wallet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-30 19:05:45 -05:00
DanSandClaude Opus 5 da191e447d Merge branch 'dev' for the v1.3.0 release 
72 commits since 1c3523aac (2026-07-23). Highlights:

  consensus  BLOCK_VALID_CONTEXT no longer overwrites the block validity level
             (6d282db21); dead CBOPRET coinbase price check removed (267e6f7ad);
             PROTOCOL_VERSION 2000000 -> 2000001, MIN_PEER_PROTO_VERSION held.
  sync       DRAGONX checkpoints extended to 3,226,000 -- past the RandomX
             activation at 2,838,976, so the skip-below-checkpoint path finally
             fires (7dc904c96).
  wallet     interrupted rescans no longer hide funds and no longer latch
             fAbortRescan (a6b6f80db); rescan progress is checkpointed when a scan
             is interrupted (c1040028e); salvaged wallets open in a degraded mode
             with hdchain-mismatch detection (5634aed75).
  stratum    stratummine gated behind an explicit flag and hardened (db42091ce);
             low-difficulty shares rejected before spending a RandomX hash
             (fa16e740b).
  release    version bumped to v1.3.0 (af7d9e230) and doc/release-process.md
             corrected -- it previously omitted signing entirely, which is why
             v1.1.0 and v1.2.0 were tagged but never became installable releases.

Soak before merge: ~86 node-hours on c1040028e across all seven seeds, plus the
primary -- the pool's block-template source -- running it in production with the
pool accepting shares. Fleet consensus verified at a common height throughout.

Not included, deliberately: branch audit-fixes-20260901 (two commits, both marked
UNTESTED) stays parked. The ported rpc-tests have never been run green and are not
a gate for this release.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
 2026-09-01 19:09:04 -05:00
DanSandClaude Opus 5 fc26e2e0a0 doc: stamp the man pages with the release version, not a build sha 
The man pages were regenerated at e2e10f6ef from a binary built at af7d9e230 and
carry "v1.3.0-af7d9e230". The release will be tagged v1.3.0, so the shipped man
pages would contradict the tag they ship under -- the same provenance gap af7d9e230
itself was written to close.

A faithful regeneration cannot fix this before the tag exists: gen-manpages.sh reads
--version from the binary, genbuild.sh derives that from `git describe`, and that
only prints exactly "v1.3.0" once the annotated tag is in place. Regenerating after
tagging would mean committing on top of the tag and moving it. doc/release-process.md
anticipates this and documents hardcoding the version instead.

Content is untouched and needs no regeneration: all 150 daemon options in the man
page match `dragonxd --help` exactly, including the five -autoshield* flags.

Not changed: dragonx-tx.1 still describes itself as "hush-tx utility". That string
comes from the binary's own --help, so editing the man page would make it diverge
from what the tool actually prints. It is a binary-level branding fix, not a doc one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
 2026-09-01 19:06:54 -05:00
DanSandClaude Opus 5 1d2d5f32d0 version: bump PROTOCOL_VERSION for the v1.3.0 consensus changes 
doc/release-process.md requires PROTOCOL_VERSION to increase by 1 for any release
carrying a consensus change. This one does: 6d282db21 stops BLOCK_VALID_CONTEXT
overwriting the block validity level, 267e6f7ad drops the dead CBOPRET price
validation from the coinbase check, and pow.cpp is 157 lines lighter. The value
has not moved since 85c8d7f7d in March.

MIN_PEER_PROTO_VERSION stays at 2000000. It currently equals PROTOCOL_VERSION, so
raising it in step would disconnect every peer still announcing 2000000 -- which is
every user, because no DragonX release has ever been installable: the wallet's
updater pins an ed25519 key and sets kDaemonRequireSignature = true, and no release
has ever shipped a .sig. Bumping only the advertised version lets 1.3.0 nodes be
identified on the wire without partitioning the network on release day.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
 2026-09-01 19:06:54 -05:00
52 changed files with 6922 additions and 2429 deletions
Expand all files Collapse all files
doc/man/dragonx-cli.1
+1 -1
View File
@@ -3,7 +3,7 @@
.SH NAME .SH NAME
dragonx-cli \- manual page for dragonx-cli v1.3.0 dragonx-cli \- manual page for dragonx-cli v1.3.0
.SH DESCRIPTION .SH DESCRIPTION
DragonX RPC client version v1.3.0\-af7d9e230 DragonX RPC client version v1.3.0
.PP .PP
In order to ensure you are adequately protecting your privacy when using In order to ensure you are adequately protecting your privacy when using
DragonX, please see <https://dragonx.is/security/>. DragonX, please see <https://dragonx.is/security/>.
doc/man/dragonx-tx.1
+1 -1
View File
@@ -3,7 +3,7 @@
.SH NAME .SH NAME
dragonx-tx \- manual page for dragonx-tx v1.3.0 dragonx-tx \- manual page for dragonx-tx v1.3.0
.SH DESCRIPTION .SH DESCRIPTION
hush\-tx utility version v1.3.0\-af7d9e230 hush\-tx utility version v1.3.0
.SS "Usage:" .SS "Usage:"
.TP .TP
hush\-tx [options] <hex\-tx> [commands] hush\-tx [options] <hex\-tx> [commands]
doc/man/dragonxd.1
+1 -1
View File
@@ -3,7 +3,7 @@
.SH NAME .SH NAME
dragonxd \- manual page for dragonxd v1.3.0 dragonxd \- manual page for dragonxd v1.3.0
.SH DESCRIPTION .SH DESCRIPTION
DragonX Daemon version v1.3.0\-af7d9e230 DragonX Daemon version v1.3.0
.PP .PP
In order to ensure you are adequately protecting your privacy when using In order to ensure you are adequately protecting your privacy when using
DragonX, please see <https://dragonx.is/security/>. DragonX, please see <https://dragonx.is/security/>.
qa/fd-guard/.gitignore
+4
View File
@@ -0,0 +1,4 @@
shim/libfdguard.so
shim/fdguard_test
*.o
shim/t/
qa/fd-guard/README.md
+55
View File
@@ -0,0 +1,55 @@
# fd-guard: catch stray close() of files a thread doesn't own
This is the harness that found the wolfSSL `CloseSocket` macro bug. With that bug, P2P disconnects closed their fd number a second time, after another thread had reused it, and BDB then wrote wallet pages into `blocks/*.dat`.
It has two parts:
- `shim/`: an `LD_PRELOAD` guard around `close`, `fclose`, `dup2` and `dup3`.
- Before each close it resolves `/proc/self/fd/N`.
- It logs a backtrace when a non-owner closes `wallet.dat`, `database/log.*` or `blocks/*.dat`, and for every close that fails with `EBADF` (the second half of a double close).
- `FDGUARD_ABORT=1` makes the first hit abort.
- `repro/`: isolated regtest runs that put a shimmed `dragonxd` under load:
- **rpc:** impatient and RST clients, half-open connections, work-queue bursts and long-polls;
- **p2p:** TLS peers churned by `disconnectnode`/ban/kill, plus misbehaving raw TCP/TLS clients;
- **tor:** a fake control port;
- **mining:** with `blocknotify` and autoshield.
Each run executes in its own unprivileged network namespace: loopback only, no route.
## Requirements
- Linux.
- `unshare -rn` (unprivileged user namespaces), `nsenter`, `ip`, python3, gcc, and `addr2line` for symbolizing.
- A `dragonxd`/`dragonx-cli` built with `-g`. Adding `-rdynamic` makes backtraces readable without `addr2line`.
- A regtest datadir with a grown wallet to copy from (`GOLDEN`). It is never modified.
## Running
```sh
qa/fd-guard/shim/build.sh # builds libfdguard.so
qa/fd-guard/shim/run_tests.sh # self-test: double close, foreign close of a guarded file
cd qa/fd-guard/repro
export GOLDEN=/path/to/regtest/datadir BIN=/path/to/build/src # RUNS defaults to ~/fd-guard-runs
LOADS=p2pmin,mine DISC_INT=5 GUARD=all ./go.sh p2pmin 300 # 1 peer, disconnectnode every 5 s
LOADS=rpc,p2p,tor,mine WQ=64 RT=8 GUARD=all ./go.sh full 1200 # everything, production-like work queue
python3 tools/tally.py "$RUNS/<name>/fdguard.log" # who closed what
python3 ../shim/symbolize.py --summary "$RUNS/<name>/fdguard.log"
python3 tools/scan_bdb.py "$RUNS/<name>"/target/regtest/blocks/*.dat # BDB pages inside block files
```
Each run writes `run.txt` (full log plus summary), `fdguard.log` and the target's `debug.log` and `db.log` under `$RUNS/<name>/`.
## What to expect
Measured on a 2041-note regtest wallet, before and after the fix (`net: stop wolfSSL's CloseSocket macro ...`), under the same loads:
| Run | Binary | EBADF closes | Non-owner closes (wallet/log/blk) | libevent `Epoll` failures | `db.log` |
|---|---|---|---|---|---|
| `p2pmin`, 300 s | before | 51 (≈ 1 per disconnect) | 1 | 0 | empty |
| | after | 0 | 0 | 0 | empty |
| `full`, 1200 s, `WQ=64 RT=8` | before | 930 | 186 (37) | 111 | 632 KB: 7500 × `BDB0060 PANIC`, `write failed for page N` |
| | after | 0 | 0 | 0 | empty |
- RPC-only and Tor-only loads give 0 reports, even before the fix: P2P disconnects are the only origin.
- `scan_bdb.py` found no BDB pages in block files in any run. The landing of a page in a block file needs the reused number to be a writable block file at the moment BDB writes. Production did hit that (blk00011/blk00020), but it is rare.
- `SSL_write ... closing connection` lines are expected under `tcpchurn`. They are how the load triggers disconnects, not a symptom.
qa/fd-guard/repro/common.sh
+72
View File
@@ -0,0 +1,72 @@
# shellcheck shell=bash
# Shared config + helpers for the fd-double-close REPRODUCTION harness.
# Regtest only. Meant to run INSIDE an unprivileged network namespace (see ns.sh),
# so every port is private to the namespace and nothing can reach a real network.
set +e
REPRO=${REPRO:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)}
SRCROOT=$(cd "$REPRO/../../.." && pwd)
BIN=${BIN:-$SRCROOT/src} # dir with dragonxd + dragonx-cli (build with -g -rdynamic for backtraces)
DRAGONXD=$BIN/dragonxd
CLI=$BIN/dragonx-cli
RUNS=${RUNS:-$HOME/fd-guard-runs} # run output (block files, logs): keep it out of the repo
GOLDEN=${GOLDEN:?set GOLDEN to a regtest datadir with a grown wallet (it is only ever copied)}
ASMAP=${ASMAP:-$SRCROOT/asmap.dat}
SP=${SP:-$HOME/.zcash-params/sapling-spend.params}
OP=${OP:-$HOME/.zcash-params/sapling-output.params}
SHIM=${SHIM:-$REPRO/../shim/libfdguard.so}
RPCUSER=ab
RPCPASS=abpass
RPC_T=18901 ; P2P_T=18911
RPC_H1=18902; P2P_H1=18912
RPC_H2=18903; P2P_H2=18913
TORPORT=18950
clin() { local dd=$1 rp=$2; shift 2; "$CLI" -regtest -datadir="$dd" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$rp "$@"; }
write_conf() { printf 'rpcuser=%s\nrpcpassword=%s\nrpcport=%s\n' "$RPCUSER" "$RPCPASS" "$2" > "$1/DRAGONX.conf"; }
# prep_golden <dd> : copy of the golden wallet datadir (2041 notes, txindex=1, height 286)
prep_golden() {
local dd=$1
rm -rf "$dd"; cp -a "$GOLDEN" "$dd"
cp -f "$ASMAP" "$dd/asmap.dat"; cp -f "$SP" "$dd/sapling-spend.params"; cp -f "$OP" "$dd/sapling-output.params"
write_conf "$dd" "$RPC_T"
: > "$dd/regtest/debug.log"; rm -f "$dd/regtest/db.log"
}
# prep_fresh <dd> <rpcport> : empty regtest datadir (helpers)
prep_fresh() {
local dd=$1 rp=$2
rm -rf "$dd"; mkdir -p "$dd"
cp -f "$ASMAP" "$dd/asmap.dat"; cp -f "$SP" "$dd/sapling-spend.params"; cp -f "$OP" "$dd/sapling-output.params"
write_conf "$dd" "$rp"
}
start_helper() { # <dd> <p2p> <rpc> <logfile>
local dd=$1 p2p=$2 rp=$3 lg=$4
( cd "$dd" && exec nice -n 10 "$DRAGONXD" -regtest -datadir="$dd" -testnode=1 -checkpoints=0 \
-listen=1 -bind=127.0.0.1 -port=$p2p -connect=127.0.0.1:1 -addnode=127.0.0.1:$P2P_T \
-dnsseed=0 -dns=0 -asmap="$dd/asmap.dat" -autoshield=0 \
-rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$rp -listenonion=0 -maxinboundfromip=100 \
-debug=net -debug=tls -daemon ) >> "$lg" 2>&1
}
waitrpc() { # <dd> <rpc> [secs]
local dd=$1 rp=$2 n=${3:-300} i
for i in $(seq 1 "$n"); do clin "$dd" "$rp" getblockcount >/dev/null 2>&1 && return 0; sleep 1; done
return 1
}
stop_daemon() { # <dd> <rpc> (helpers, started with -daemon)
local dd=$1 rp=$2 pid i
pid=$(cat "$dd/regtest/hushd.pid" 2>/dev/null)
clin "$dd" "$rp" stop >/dev/null 2>&1
for i in $(seq 1 90); do { [ -z "$pid" ] || ! kill -0 "$pid" 2>/dev/null; } && return 0; sleep 1; done
[ -n "$pid" ] && kill -9 "$pid" 2>/dev/null
}
MARKERS=('libevent: Epoll' 'Socket close failed' 'Bad file descriptor' 'Operation not permitted' 'Illegal seek'
'SSL_write' 'Unable to seek' 'socket select error' 'write failed for page' 'PANIC' 'error scanning DB'
'terminate' 'Assertion' 'EXCEPTION')
qa/fd-guard/repro/go.sh
Executable
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
# go.sh <name> <seconds> -- create a private network namespace (loopback only, no route),
# run run.sh inside it, then tear the namespace down. Env knobs pass through to run.sh.
export REPRO=${REPRO:-$(cd "$(dirname "$0")" && pwd)}
export RUNS=${RUNS:-$HOME/fd-guard-runs}
OUT=$RUNS/$1; mkdir -p "$OUT"
setsid unshare -rn sh -c "ip link set lo up && echo \$\$ > $OUT/ns.pid && exec sleep infinity" </dev/null >/dev/null 2>&1 &
for _ in 1 2 3 4 5 6 7 8 9 10; do [ -s "$OUT/ns.pid" ] && break; sleep 0.3; done
NSP=$(cat "$OUT/ns.pid")
cp "$REPRO/run.sh" "$OUT/run.sh.snapshot" # run a frozen copy: editing run.sh cannot disturb a live run
nsenter -t "$NSP" -U -n --preserve-credentials bash "$OUT/run.sh.snapshot" "$@" > "$OUT/go.txt" 2>&1
rc=$?
kill "$NSP" 2>/dev/null
echo "go.sh: run.sh rc=$rc, namespace holder $NSP killed" >> "$OUT/go.txt"
qa/fd-guard/repro/run.sh
Executable
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env bash
# run.sh <name> <seconds> -- run INSIDE the network namespace (go.sh does that).
# Env knobs:
# LOADS=rpc,p2p,tor,mine which load generators run (default all four); p2pmin = 1 helper + disconnectnode every DISC_INT s
# WQ=4 RT=2 -rpcworkqueue / -rpcthreads of the target
# GUARD=all FDGUARD_GUARD (db,blk | all)
# SHIMON=1 run the target under LD_PRELOAD=libfdguard.so
# STRACE=0 1 = also wrap the target in strace (failed fd syscalls only)
# TRACE=0 1 = FDGUARD_TRACE_ALL (one line per open/close; big)
# MINE_INT=3 seconds between generate 1
# HAMMER_ARGS=... override the hammer thread mix
# STOP_ON_HIT=0 1 = end the run 60 s after the first NON-OWNER db/blk close
# EXTRA=... extra target args
source "$REPRO/common.sh"
NAME=$1; DUR=${2:-600}
OUT=$RUNS/$NAME
LOADS=${LOADS:-rpc,p2p,tor,mine}; WQ=${WQ:-4}; RT=${RT:-2}; GUARD=${GUARD:-all}
SHIMON=${SHIMON:-1}; STRACE=${STRACE:-0}; TRACE=${TRACE:-0}; MINE_INT=${MINE_INT:-3}
STOP_ON_HIT=${STOP_ON_HIT:-0}
HAMMER_ARGS=${HAMMER_ARGS:---normal 8 --impatient 8 --slowfire 2 --halfopen 3 --pipeline 3 --garbage 2 --slowread 2 --longpoll 4 --burst-size 16 --burst-every 2}
has() { case ",$LOADS," in *",$1,"*) return 0;; esac; return 1; }
mkdir -p "$OUT/logs"; rm -f "$OUT/STOP"
DD_T=$OUT/target; DD_H1=$OUT/helper1; DD_H2=$OUT/helper2
exec > >(tee -a "$OUT/run.txt") 2>&1
echo "=== repro run $NAME $(date -u +%FT%TZ) dur=${DUR}s loads=$LOADS wq=$WQ rt=$RT guard=$GUARD shim=$SHIMON strace=$STRACE"
echo "netns check (must be ENETUNREACH=101): $(python3 -c 'import socket;print(socket.socket().connect_ex(("1.1.1.1",80)))')"
echo "binary: $DRAGONXD build-id: $(readelf -n "$DRAGONXD" | awk '/Build ID/{print $3}')"
echo "dragonxd already running in this namespace? $(pgrep -a -x dragonxd | wc -l) (other agents' nodes live in other namespaces)"
PIDS=()
cleanup() {
echo "--- cleanup $(date -u +%T)"
touch "$OUT/STOP"
for p in "${PIDS[@]}"; do kill "$p" 2>/dev/null; done
stop_daemon "$DD_H2" "$RPC_H2"; stop_daemon "$DD_H1" "$RPC_H1"
if [ -n "$TPID" ] && kill -0 "$TPID" 2>/dev/null; then
timeout 20 "$CLI" -regtest -datadir="$DD_T" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T stop >/dev/null 2>&1
for _ in $(seq 1 60); do kill -0 "$TPID" 2>/dev/null || break; sleep 1; done
# abandoned longpoll getblocktemplate calls can hold every -rpcthreads worker, so RPC stop may never run:
# fall back to SIGTERM (same clean StartShutdown path), SIGKILL only as a last resort
kill -0 "$TPID" 2>/dev/null && { echo "RPC stop did not finish in 60 s -> SIGTERM"; kill -TERM "$TPID"; }
for _ in $(seq 1 180); do kill -0 "$TPID" 2>/dev/null || break; sleep 1; done
kill -0 "$TPID" 2>/dev/null && { echo "target did not stop -> SIGKILL"; kill -9 "$TPID"; }
fi
[ -n "$WPID" ] && wait "$WPID" 2>/dev/null; echo "target exit status: $?"
for p in "${PIDS[@]}"; do kill -9 "$p" 2>/dev/null; done
summarize
}
summarize() {
local DL=$DD_T/regtest/debug.log
echo "=== markers in target debug.log"
for m in "${MARKERS[@]}"; do printf ' [%6d] %s\n' "$(grep -c -- "$m" "$DL" 2>/dev/null)" "$m"; done
echo " db.log bytes: $(stat -c %s "$DD_T/regtest/db.log" 2>/dev/null || echo absent)"
grep -q 'Shutdown: done' "$DL" && echo " clean shutdown: yes" || echo " clean shutdown: NO"
if [ "$SHIMON" = 1 ]; then
echo "=== fdguard reports"; grep -o '=== FDGUARD #[0-9]* [A-Z-]*' "$OUT/fdguard.log" | awk '{print $4}' | sort | uniq -c
grep '^S ' "$OUT/fdguard.log" | tail -1
python3 "$REPRO/../shim/symbolize.py" --summary "$OUT/fdguard.log" > "$OUT/fdguard.summary.txt" 2>&1
head -60 "$OUT/fdguard.summary.txt"
fi
echo "=== BDB-page-in-blockfile detector"
python3 "$REPRO/tools/scan_bdb.py" "$DD_T"/regtest/blocks/blk*.dat "$DD_T"/regtest/blocks/rev*.dat 2>&1 | tail -15
echo "=== end $(date -u +%FT%TZ)"
}
trap cleanup EXIT
trap 'exit 130' INT TERM
echo "--- prep datadirs"
prep_golden "$DD_T"
prep_fresh "$DD_H1" "$RPC_H1"; prep_fresh "$DD_H2" "$RPC_H2"
cat > "$OUT/blocknotify.sh" <<EOF
#!/bin/sh
# pool-like blocknotify: a couple of dragonx-cli calls per block
$CLI -regtest -datadir=$DD_T -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T getblock "\$1" >/dev/null 2>&1
$CLI -regtest -datadir=$DD_T -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T getblocktemplate >/dev/null 2>&1
echo "\$(date +%s) \$1" >> $OUT/logs/blocknotify.log
EOF
chmod +x "$OUT/blocknotify.sh"
if has tor; then
nice -n 10 python3 "$REPRO/tools/faketor.py" --port $TORPORT > "$OUT/logs/faketor.txt" 2>&1 & PIDS+=($!)
TORARGS="-listenonion=1 -torcontrol=127.0.0.1:$TORPORT -debug=tor"
else
TORARGS="-listenonion=0"
fi
if has p2p || has p2pmin; then P2PARGS="-listen=1 -bind=127.0.0.1 -port=$P2P_T -allowlist=127.0.0.0/8 -maxinboundfromip=100"
else P2PARGS="-listen=0 -port=$P2P_T"; fi
TARGS=(-regtest -datadir="$DD_T" -testnode=1 -checkpoints=0 $P2PARGS -connect=127.0.0.1:1 -dnsseed=0 -dns=0
-asmap="$DD_T/asmap.dat" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T
-txindex=1 -autoshield=1 -rpcworkqueue=$WQ -rpcthreads=$RT
"-blocknotify=$OUT/blocknotify.sh %s" $TORARGS -debug=net -debug=tls -printtoconsole=0 -daemon=0 $EXTRA)
echo "target args: ${TARGS[*]}" | sed "s/$RPCPASS/***/g"
ENVV=()
if [ "$SHIMON" = 1 ]; then
ENVV=(FDGUARD_LOG="$OUT/fdguard.log" FDGUARD_GUARD="$GUARD" FDGUARD_ABORT=0 FDGUARD_STATS_SEC=60 FDGUARD_MAXREPORTS=2000 LD_PRELOAD="$SHIM")
[ "$TRACE" = 1 ] && ENVV+=(FDGUARD_TRACE_ALL=1)
fi
WRAP=()
[ "$STRACE" = 1 ] && WRAP=(strace -f -tt -qq -e trace=close,openat,accept4,socket,dup2,dup3 -e status=failed -o "$OUT/strace.txt")
( cd "$DD_T" && exec env "${ENVV[@]}" nice -n 10 "${WRAP[@]}" "$DRAGONXD" "${TARGS[@]}" ) > "$OUT/logs/target_stdout.txt" 2>&1 &
WPID=$!
echo "--- target launched (wrapper pid $WPID); waiting for RPC"
waitrpc "$DD_T" "$RPC_T" 600 || { echo "target never answered RPC"; exit 1; }
TPID=$(cat "$DD_T/regtest/hushd.pid" 2>/dev/null); [ -z "$TPID" ] && TPID=$WPID
echo "target up pid=$TPID height=$(clin "$DD_T" "$RPC_T" getblockcount) $(date -u +%T)"
T0=$(date +%s)
if has p2p; then
start_helper "$DD_H1" "$P2P_H1" "$RPC_H1" "$OUT/logs/helper1.txt"
start_helper "$DD_H2" "$P2P_H2" "$RPC_H2" "$OUT/logs/helper2.txt"
waitrpc "$DD_H1" "$RPC_H1" 120; waitrpc "$DD_H2" "$RPC_H2" 120
nice -n 10 python3 "$REPRO/tools/tcpchurn.py" --port $P2P_T --duration "$DUR" --concurrency 8 > "$OUT/logs/tcpchurn.txt" 2>&1 & PIDS+=($!)
nice -n 10 bash "$REPRO/tools/rpcchurn.sh" "$OUT" "$DUR" > "$OUT/logs/rpcchurn.txt" 2>&1 & PIDS+=($!)
fi
if has p2pmin; then
# minimal P2P: ONE well-behaved helper peer, and an ordinary "disconnectnode" every DISC_INT s
# (the helper's -addnode re-dials the target). No raw TCP/TLS churn, no bans, no kills.
start_helper "$DD_H1" "$P2P_H1" "$RPC_H1" "$OUT/logs/helper1.txt"; waitrpc "$DD_H1" "$RPC_H1" 120
( k=0; while [ ! -e "$OUT/STOP" ]; do sleep "${DISC_INT:-5}"; for a in $(clin "$DD_T" "$RPC_T" getpeerinfo 2>/dev/null | python3 -c 'import json,sys; [print(p["addr"]) for p in json.load(sys.stdin)]' 2>/dev/null); do
clin "$DD_T" "$RPC_T" disconnectnode "$a" >/dev/null 2>&1 && { k=$((k+1)); echo "$(date -u +%T.%N) disconnectnode $a" >> "$OUT/logs/disconnects.log"; }; done
sleep 1; clin "$DD_H1" "$RPC_H1" addnode "127.0.0.1:$P2P_T" onetry >/dev/null 2>&1 # helper re-dials now (its -addnode retry is ~2 min)
echo "$k" > "$OUT/logs/disconnects.txt"; done ) & PIDS+=($!)
fi
if has rpc; then
MINEARG=""; has mine && MINEARG="--mine --mine-interval $MINE_INT"
nice -n 10 python3 "$REPRO/tools/hammer.py" --port $RPC_T --user $RPCUSER --password $RPCPASS --duration "$DUR" \
$HAMMER_ARGS $MINEARG --pid "$TPID" --logfile "$OUT/logs/hammer.log" > "$OUT/logs/hammer_stdout.txt" 2>&1 & PIDS+=($!)
elif has mine; then
( while [ ! -e "$OUT/STOP" ]; do clin "$DD_T" "$RPC_T" generate 1 >/dev/null 2>&1; sleep "$MINE_INT"; done ) & PIDS+=($!)
fi
# ---- poll
end=$(( T0 + DUR )); n=0; hit_at=0
while [ "$(date +%s)" -lt "$end" ] && [ ! -e "$OUT/STOP" ]; do
sleep 30; n=$((n+1))
alive=no; kill -0 "$TPID" 2>/dev/null && alive=yes
DL=$DD_T/regtest/debug.log
h=$(clin "$DD_T" "$RPC_T" getblockcount 2>/dev/null); c=$(clin "$DD_T" "$RPC_T" getconnectioncount 2>/dev/null)
rep=$(grep -c '^=== FDGUARD' "$OUT/fdguard.log" 2>/dev/null); noc=$(grep -c 'NON-OWNER-CLOSE' "$OUT/fdguard.log" 2>/dev/null)
ebf=$(grep -c 'EBADF-CLOSE' "$OUT/fdguard.log" 2>/dev/null); epl=$(grep -c '=== FDGUARD #[0-9]* EPOLL' "$OUT/fdguard.log" 2>/dev/null)
dbnoc=$(grep -A3 'NON-OWNER-CLOSE' "$OUT/fdguard.log" 2>/dev/null | grep -c 'class=\(db\|blk\)')
printf '[poll %3d %s +%4ds] alive=%s h=%s conns=%s | fdguard reports=%s nonowner=%s(db/blk %s) ebadf=%s epoll=%s | Epoll=%s SockCloseFail=%s EBADF=%s EPERM=%s SSL_write=%s seek=%s sel=%s | db.log=%s\n' \
$n "$(date -u +%T)" $(( $(date +%s) - T0 )) $alive "$h" "$c" "$rep" "$noc" "$dbnoc" "$ebf" "$epl" \
"$(grep -c 'libevent: Epoll' "$DL")" "$(grep -c 'Socket close failed' "$DL")" "$(grep -c 'Bad file descriptor' "$DL")" \
"$(grep -c 'Operation not permitted' "$DL")" "$(grep -c 'SSL_write' "$DL")" "$(grep -c 'Unable to seek' "$DL")" \
"$(grep -c 'socket select error' "$DL")" "$(stat -c %s "$DD_T/regtest/db.log" 2>/dev/null)" | tee -a "$OUT/poll.txt"
[ "$alive" = no ] && { echo "!!! target died"; break; }
if [ "$STOP_ON_HIT" = 1 ] && [ "${dbnoc:-0}" -gt 0 ]; then
[ "$hit_at" = 0 ] && hit_at=$(date +%s)
[ $(( $(date +%s) - hit_at )) -ge 60 ] && { echo "--- stop-on-hit"; break; }
fi
done
cat "$OUT/logs/tcpchurn.txt" "$OUT/logs/rpcchurn.txt" 2>/dev/null
grep -E 'TOTAL|totals' -A12 "$OUT/logs/hammer.log" 2>/dev/null | tail -14
qa/fd-guard/repro/tools/faketor.py
Executable
+98
View File
@@ -0,0 +1,98 @@
#!/usr/bin/env python3
"""
Fake Tor control port for the fd-bug harness.
dragonxd's torcontrol.cpp opens a libevent bufferevent to -torcontrol=<host:port>,
speaks the control protocol (PROTOCOLINFO / AUTHCHALLENGE / AUTHENTICATE / ADD_ONION),
and on EOF/ERROR reconnects with exponential backoff. We give it a server on a
loopback port that behaves badly in several ways so the reconnect/teardown loop
(bufferevent_free with BEV_OPT_CLOSE_ON_FREE, the event_base, the reconnect timer)
runs constantly. Each accepted connection picks a behaviour at random:
close : accept, then immediately close (EOF right away)
rst : accept, set SO_LINGER 0, close -> sends RST
stall : accept, read nothing, hold the socket open then drop it
garbage : send non-control-protocol bytes then close
halfok : answer PROTOCOLINFO plausibly, then drop before AUTHENTICATE completes
Loopback only. Run with --port. Ctrl-C / SIGTERM stops it.
"""
import argparse, random, socket, struct, threading, time, sys, signal
STOP = threading.Event()
def handle(conn, mode):
try:
conn.settimeout(5.0)
if mode == "close":
conn.close(); return
if mode == "rst":
conn.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER,
struct.pack("ii", 1, 0))
conn.close(); return
if mode == "stall":
# read whatever the controller sends but never reply; hold 2-4s
try:
conn.recv(4096)
except Exception:
pass
time.sleep(random.uniform(2.0, 4.0))
conn.close(); return
if mode == "garbage":
try:
conn.recv(4096)
except Exception:
pass
conn.sendall(b"\x00\xff\x01garbage not a tor reply\r\n" * 4)
conn.close(); return
if mode == "halfok":
# read the PROTOCOLINFO line, give a plausible 250 reply advertising
# NULL auth, then drop the connection before AUTHENTICATE finishes.
try:
conn.recv(4096)
except Exception:
pass
conn.sendall(b"250-PROTOCOLINFO 1\r\n"
b"250-AUTH METHODS=NULL\r\n"
b'250-VERSION Tor="0.4.7.13"\r\n'
b"250 OK\r\n")
time.sleep(random.uniform(0.2, 1.0))
conn.close(); return
except Exception:
try: conn.close()
except Exception: pass
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--port", type=int, required=True)
ap.add_argument("--host", default="127.0.0.1")
args = ap.parse_args()
modes = ["close", "rst", "stall", "garbage", "halfok"]
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind((args.host, args.port))
srv.listen(16)
srv.settimeout(1.0)
print(f"faketor: listening on {args.host}:{args.port}", flush=True)
def onsig(*_): STOP.set()
signal.signal(signal.SIGTERM, onsig)
signal.signal(signal.SIGINT, onsig)
n = 0
while not STOP.is_set():
try:
conn, _ = srv.accept()
except socket.timeout:
continue
except Exception:
break
mode = modes[n % len(modes)]
n += 1
threading.Thread(target=handle, args=(conn, mode), daemon=True).start()
srv.close()
print(f"faketor: stopped after {n} connections", flush=True)
if __name__ == "__main__":
main()
qa/fd-guard/repro/tools/hammer.py
Executable
+796
View File
@@ -0,0 +1,796 @@
#!/usr/bin/env python3
# hammer.py -- JSON-RPC robustness/load harness for a LOCAL regtest dragonxd.
#
# Scope: connects only to the --host/--port you pass (meant to be 127.0.0.1 and a
# regtest RPC port). Its job is to reproduce a known file-descriptor bug in the
# libevent-2.1.8 HTTP RPC server (src/httpserver.cpp) by exercising the same
# request patterns the production pool box sees: many concurrent clients, clients
# that disconnect mid-request/mid-response (the explorer indexer's 120 s client
# timeout), work-queue overflow, keep-alive pipelining, and malformed input.
# stdlib only.
#
# Correctness oracle: every request carries a unique JSON-RPC id; if a reply comes
# back with a different id on the same socket, that is cross-talk (the server wrote
# one client's reply onto another client's fd) and is logged + counted.
#
# See README.txt for the exact invocation.
import argparse
import base64
import collections
import json
import os
import random
import socket
import struct
import threading
import time
# ------------------------------------------------------------------ config/state
class C:
host = "127.0.0.1"
port = 19461
auth = ""
ctimeout = 15.0
CFG = C()
STOP = threading.Event()
SHARED = {"hashes": [], "lpid": None, "zaddrs": [], "rawblock": None}
SHARED_LOCK = threading.Lock()
_ID = [random.SystemRandom().randrange(1 << 40)]
_ID_LOCK = threading.Lock()
LOGLOCK = threading.Lock()
LOGF = [None]
def next_id():
with _ID_LOCK:
_ID[0] += 1
return _ID[0]
def log(msg):
line = "%s %s" % (time.strftime("%H:%M:%S"), msg)
with LOGLOCK:
print(line, flush=True)
if LOGF[0]:
LOGF[0].write(line + "\n"); LOGF[0].flush()
# ------------------------------------------------------------------ stats
KEYS = ["sent", "ok", "rpcerr", "wq500", "http401", "http4xx", "httpother",
"early_rst", "early_fin", "srvreset", "srveof", "ctimeout", "connfail",
"idmismatch", "malformed", "mined", "half_open", "half_srvclose"]
class Stats:
def __init__(self):
self.lk = threading.Lock()
self.total = collections.Counter()
self.win = collections.Counter()
self.anom = []
def inc(self, k, n=1):
with self.lk:
self.total[k] += n; self.win[k] += n
def anomaly(self, kind, detail):
with self.lk:
if len(self.anom) < 300:
self.anom.append((time.time(), kind, detail))
log("ANOMALY %s %s" % (kind, detail))
def swap(self):
with self.lk:
w = self.win; self.win = collections.Counter(); return w
ST = Stats()
# ------------------------------------------------------------------ socket helpers
def auth_header(u, p):
return "Basic " + base64.b64encode(("%s:%s" % (u, p)).encode()).decode()
def mkconn(timeout=None, rcvbuf=None):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
if rcvbuf:
s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, rcvbuf)
s.settimeout(timeout if timeout is not None else CFG.ctimeout)
s.connect((CFG.host, CFG.port))
return s
def end_rst(s):
# linger 0 -> TCP RST on close (models a client that dies / resets abruptly)
try:
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
except OSError:
pass
try:
s.close()
except OSError:
pass
def end_fin(s):
try:
s.shutdown(socket.SHUT_RDWR)
except OSError:
pass
try:
s.close()
except OSError:
pass
def end_early(s, how=None):
how = how or random.choice(("rst", "fin"))
if how == "rst":
end_rst(s); ST.inc("early_rst")
else:
end_fin(s); ST.inc("early_fin")
def req_bytes(method, params, rid, keepalive=True, auth=None, uri="/",
httpmethod="POST", body=None, clen=None):
if body is None:
body = json.dumps({"jsonrpc": "1.0", "id": rid,
"method": method, "params": params}).encode()
head = [
"%s %s HTTP/1.1" % (httpmethod, uri),
"Host: %s:%d" % (CFG.host, CFG.port),
"Authorization: %s" % (auth if auth is not None else CFG.auth),
"Content-Type: application/json",
"Content-Length: %d" % (clen if clen is not None else len(body)),
"Connection: %s" % ("keep-alive" if keepalive else "close"),
"", "",
]
return "\r\n".join(head).encode() + body
class Reader:
"""Incremental HTTP/1.1 reply reader over one socket."""
def __init__(self, s):
self.s = s; self.buf = b""
def _fill(self):
d = self.s.recv(65536)
if not d:
raise EOFError()
self.buf += d
def read(self):
while b"\r\n\r\n" not in self.buf:
self._fill()
if len(self.buf) > (1 << 20):
raise ValueError("oversize header")
head, self.buf = self.buf.split(b"\r\n\r\n", 1)
lines = head.split(b"\r\n")
p = lines[0].split(b" ", 2)
if len(p) < 2 or not p[0].startswith(b"HTTP/1."):
raise ValueError("bad status line %r" % lines[0][:80])
status = int(p[1])
hdr = {}
for ln in lines[1:]:
if b":" in ln:
k, v = ln.split(b":", 1)
hdr[k.strip().lower()] = v.strip()
if hdr.get(b"transfer-encoding", b"").lower() == b"chunked":
body = b""
while True:
while b"\r\n" not in self.buf:
self._fill()
ln, self.buf = self.buf.split(b"\r\n", 1)
n = int(ln.split(b";")[0], 16)
while len(self.buf) < n + 2:
self._fill()
body += self.buf[:n]; self.buf = self.buf[n + 2:]
if n == 0:
break
elif b"content-length" in hdr:
n = int(hdr[b"content-length"])
while len(self.buf) < n:
self._fill()
body, self.buf = self.buf[:n], self.buf[n:]
else:
try:
while True:
self._fill()
except EOFError:
pass
body, self.buf = self.buf, b""
return status, hdr, body
def account(status, body, rid, method, expect401=False):
if status == 401:
ST.inc("http401")
if not expect401:
ST.anomaly("unexpected401", (method,))
return None
if status == 500 and b"Work queue depth exceeded" in body:
ST.inc("wq500"); return None
j = None
if body[:1] in (b"{", b"["):
try:
j = json.loads(body)
except ValueError:
j = None
if j is None:
if 400 <= status < 500:
ST.inc("http4xx")
else:
ST.inc("httpother")
if status == 200:
ST.anomaly("non-json-200", (method, body[:120].decode("latin1")))
return None
if isinstance(j, dict) and rid is not None and j.get("id") != rid:
ST.inc("idmismatch")
ST.anomaly("idmismatch", {"method": method, "sent": rid, "got": j.get("id"),
"body": body[:200].decode("latin1")})
if isinstance(j, dict) and j.get("error"):
ST.inc("rpcerr")
else:
ST.inc("ok")
return j
def call(s, rd, method, params, keepalive=True, expect401=False, auth=None):
rid = next_id()
try:
s.sendall(req_bytes(method, params, rid, keepalive=keepalive, auth=auth))
ST.inc("sent")
status, hdr, body = rd.read()
except socket.timeout:
ST.inc("ctimeout"); return None, False
except (ConnectionResetError, BrokenPipeError):
ST.inc("srvreset"); return None, False
except EOFError:
ST.inc("srveof"); return None, False
except ValueError as e:
ST.inc("malformed"); ST.anomaly("malformed", (method, str(e))); return None, False
except OSError:
ST.inc("srvreset"); return None, False
j = account(status, body, rid, method, expect401)
alive = keepalive and hdr.get(b"connection", b"").lower() != b"close"
return j, alive
# ------------------------------------------------------------------ menus
def a_hash():
with SHARED_LOCK:
return random.choice(SHARED["hashes"]) if SHARED["hashes"] else None
def a_zaddr():
with SHARED_LOCK:
return random.choice(SHARED["zaddrs"]) if SHARED["zaddrs"] else None
def normal_menu():
# Mostly fast calls (sub-ms) so a well-behaved client interleaves with mining.
# Heavy wallet calls (z_gettotalbalance ~2.3s on the golden wallet) appear but
# are rare, so they don't permanently peg the small worker pool on their own.
fast = [("getinfo", []), ("getblockcount", []), ("getbestblockhash", []),
("getrawmempool", []), ("getmininginfo", []), ("getdifficulty", []),
("getblockchaininfo", []), ("getpeerinfo", []),
("getblocktemplate", []), ("getnetworkinfo", [])]
h = a_hash()
if h:
fast.append(("getblock", [h, 2]))
fast.append(("getblockheader", [h]))
# "slow" here is only mildly-larger fast calls; the genuinely heavy wallet calls
# live in the paced slow-fire class so a normal dashboard poller never pegs the
# worker pool.
slow = [("coinsupply", []), ("listtransactions", ["*", 10]),
("listunspent", [0]), ("getblock", [a_hash() or "", 2])]
return random.choice(slow) if random.random() < 0.1 else random.choice(fast)
def medium_menu():
# Fast to compute (~4 ms on this node) but a non-trivial reply to write, so they
# are good targets for a mid-response disconnect. The fd-reuse race fires on the
# disconnect, not on call duration, so firing these at a HIGH rate drives the
# race hard while the worker frees in milliseconds (mining keeps its slots).
m = [("listtransactions", ["*", 10000]), ("getrawmempool", []),
("getrawmempool", [True]), ("getblockchaininfo", []),
("getblocktemplate", []), ("getchaintips", []), ("getinfo", []),
("coinsupply", []), ("listunspent", [0])]
h = a_hash()
if h:
m.append(("getblock", [h, 2]))
m.append(("getblock", [h, 1]))
return random.choice(m)
def heavy_menu():
# Genuinely slow calls (z_gettotalbalance ~1.7 s, z_listunspent ~5.7 s on the
# golden wallet). Fired only by the small, PACED slow-fire class so a worker
# keeps running a multi-second call long after the client disconnected (the
# production indexer/coinsupply scenario that widens the fd-reuse window),
# without the fire-and-disconnect backlog pinning the whole pool.
m = [("z_gettotalbalance", [0]), ("z_listunspent", [0])]
z = a_zaddr()
if z:
m.append(("z_listreceivedbyaddress", [z, 0]))
return random.choice(m)
# ------------------------------------------------------------------ clients
def client_normal(cid):
while not STOP.is_set():
keepalive = random.random() < 0.7
try:
s = mkconn()
except OSError:
ST.inc("connfail"); time.sleep(0.05); continue
rd = Reader(s)
n = random.randint(1, 8) if keepalive else 1
try:
for _ in range(n):
if STOP.is_set():
break
method, params = normal_menu()
j, alive = call(s, rd, method, params, keepalive=keepalive)
if not alive:
break
finally:
(end_fin if random.random() < 0.8 else end_rst)(s)
time.sleep(random.uniform(0.01, 0.15))
def client_impatient(cid):
# High-rate fast-disconnect churn: send a (medium) request, then disconnect
# after 1-500 ms at a random phase. This is the main driver of the fd-reuse
# race -- a disconnect concurrent with the server writing the reply / re-arming
# the socket -- and the worker frees in ms so mining is unaffected.
while not STOP.is_set():
method, params = medium_menu()
rid = next_id()
body = json.dumps({"jsonrpc": "1.0", "id": rid, "method": method,
"params": params}).encode()
phase = random.choice(("mid_request", "after_send", "mid_response"))
try:
s = mkconn(timeout=2.0)
except OSError:
ST.inc("connfail"); continue
try:
if phase == "mid_request":
full = req_bytes(method, params, rid, keepalive=False, body=body)
cut = random.randint(len(full) - len(body), len(full) - 1)
s.sendall(full[:cut])
time.sleep(random.uniform(0.001, 0.5))
end_early(s); continue
s.sendall(req_bytes(method, params, rid, keepalive=False, body=body))
ST.inc("sent")
time.sleep(random.uniform(0.001, 0.5))
if phase == "mid_response":
try:
s.recv(random.randint(0, 64)) # read a few bytes, then bail
except OSError:
pass
end_early(s)
except (OSError, socket.timeout):
ST.inc("srvreset")
end_rst(s)
time.sleep(random.uniform(0.05, 0.4))
def client_slowfire(cid, pace):
# Paced indexer/coinsupply client: fire ONE genuinely-slow request, disconnect
# after 1-500 ms (so the worker runs the multi-second call with no client left
# == the production fd-reuse window), then wait `pace` s before the next so a
# fire-and-disconnect backlog cannot pin the whole worker pool. Phase: cut the
# request mid-body, or send it whole and abort while the worker is busy, or read
# a few bytes of the reply then abort.
while not STOP.is_set():
method, params = heavy_menu()
rid = next_id()
body = json.dumps({"jsonrpc": "1.0", "id": rid, "method": method,
"params": params}).encode()
phase = random.choice(("mid_request", "after_send", "mid_response"))
try:
s = mkconn(timeout=2.0)
except OSError:
ST.inc("connfail"); time.sleep(pace); continue
try:
if phase == "mid_request":
full = req_bytes(method, params, rid, keepalive=False, body=body)
cut = random.randint(len(full) - len(body), len(full) - 1)
s.sendall(full[:cut])
time.sleep(random.uniform(0.001, 0.5)); end_early(s)
else:
s.sendall(req_bytes(method, params, rid, keepalive=False, body=body))
ST.inc("sent")
time.sleep(random.uniform(0.001, 0.5))
if phase == "mid_response":
try:
s.recv(random.randint(0, 64))
except OSError:
pass
end_early(s)
except (OSError, socket.timeout):
ST.inc("srvreset"); end_rst(s)
time.sleep(random.uniform(pace * 0.5, pace * 1.5))
def client_half_open(cid):
# connect, send nothing / partial headers, trickle, then vanish
while not STOP.is_set():
try:
s = mkconn(timeout=1.0)
except OSError:
ST.inc("connfail"); time.sleep(0.1); continue
ST.inc("half_open")
style = random.choice(("silent", "partial_headers", "full_headers_no_body"))
try:
if style == "partial_headers":
s.sendall(b"POST / HTTP/1.1\r\nHost: x\r\n")
elif style == "full_headers_no_body":
s.sendall(b"POST / HTTP/1.1\r\nHost: x\r\nAuthorization: " +
CFG.auth.encode() + b"\r\nContent-Length: 1000\r\n\r\n")
deadline = time.monotonic() + random.uniform(0.1, 20.0)
while time.monotonic() < deadline and not STOP.is_set():
time.sleep(random.uniform(0.2, 1.5))
try:
s.sendall(b" ") # one byte, keep it half-alive
except OSError:
ST.inc("half_srvclose"); break
try:
s.settimeout(0.01)
if s.recv(1) == b"":
ST.inc("half_srvclose"); break
except socket.timeout:
pass
except OSError:
ST.inc("half_srvclose"); break
finally:
end_early(s)
def client_burst(cid, size, every):
while not STOP.is_set():
batch = []
for _ in range(size):
try:
s = mkconn(timeout=3.0)
except OSError:
ST.inc("connfail"); continue
batch.append(s)
for s in batch:
try:
method, params = medium_menu()
s.sendall(req_bytes(method, params, next_id(), keepalive=False))
ST.inc("sent")
except OSError:
pass
# most close immediately (before the 500/reply is written); a few read it
keep = random.sample(batch, max(0, len(batch) // 4)) if batch else []
for s in batch:
if s in keep:
continue
end_early(s)
for s in keep:
try:
rd = Reader(s); st, hd, bd = rd.read(); account(st, bd, None, "burst")
except Exception:
ST.inc("srvreset")
finally:
end_fin(s)
for _ in range(int(every * 10)):
if STOP.is_set():
break
time.sleep(0.1)
def client_pipeline(cid):
while not STOP.is_set():
try:
s = mkconn()
except OSError:
ST.inc("connfail"); time.sleep(0.05); continue
n = random.randint(2, 8)
ids = []
blob = b""
for _ in range(n):
rid = next_id(); ids.append(rid)
method, params = normal_menu()
blob += req_bytes(method, params, rid, keepalive=True)
try:
s.sendall(blob); ST.inc("sent", n)
rd = Reader(s)
toread = random.randint(0, n) # sometimes leave replies unread, then close
for k in range(toread):
st, hd, bd = rd.read()
account(st, bd, ids[k], "pipeline")
except (OSError, EOFError, ValueError):
ST.inc("srvreset")
finally:
end_early(s)
time.sleep(random.uniform(0, 0.03))
def client_garbage(cid):
probes = [
("submitblock", lambda: ["ab" * random.randint(1, 4000)]),
("submitblock", lambda: [os.urandom(random.randint(1, 200)).hex()]),
("submitblock", lambda: ["zzzz"]),
("submitblock", lambda: [""]),
]
while not STOP.is_set():
pick = random.random()
try:
s = mkconn(timeout=5.0)
except OSError:
ST.inc("connfail"); continue
try:
if pick < 0.5:
method, mk = random.choice(probes)
j, _ = call(s, Reader(s), method, mk(), keepalive=False)
elif pick < 0.65:
# malformed JSON body
s.sendall(req_bytes("x", [], next_id(), keepalive=False,
body=b'{"jsonrpc":"1.0","method":'))
ST.inc("sent")
try:
account(*((lambda r: (r[0], r[2]))(Reader(s).read())), rid=None, method="badjson")
except Exception:
ST.inc("srvreset")
elif pick < 0.8:
# unknown method / wrong URI
s.sendall(req_bytes("getinfo", [], next_id(), keepalive=False, uri="/nope"))
ST.inc("sent")
try:
st, hd, bd = Reader(s).read(); account(st, bd, None, "wronguri")
except Exception:
ST.inc("srvreset")
else:
# Content-Length larger than the body we send, then bail
body = b'{"id":1}'
s.sendall(req_bytes("getinfo", [], 1, keepalive=False, body=body, clen=len(body) + 5000))
ST.inc("sent")
time.sleep(random.uniform(0.001, 0.3))
end_early(s)
continue
except (OSError, socket.timeout):
ST.inc("srvreset")
finally:
end_early(s)
time.sleep(random.uniform(0.02, 0.2))
def client_slowread(cid):
# ask for a big reply with a tiny recv buffer and do not read it, stalling the
# server write side, then close abruptly
while not STOP.is_set():
try:
s = mkconn(timeout=3.0, rcvbuf=512)
except OSError:
ST.inc("connfail"); continue
try:
s.sendall(req_bytes("listtransactions", ["*", 10000], next_id(), keepalive=False))
ST.inc("sent")
time.sleep(random.uniform(0.05, 1.0))
except OSError:
ST.inc("srvreset")
finally:
end_early(s)
time.sleep(random.uniform(0, 0.05))
def _stale_lpid():
# A longpollid whose hash != the current tip makes getblocktemplate return at
# once (the server's wait loop condition is immediately false), so it exercises
# the long-poll request path (lpval parsing, cs_main leave/enter) and the
# mid-response disconnect WITHOUT parking a worker. On a tiny -rpcthreads pool a
# genuinely parking long-poll + RPC-driven mining self-deadlocks (both workers
# wait for a block that only a third worker could mine); that is a pool-size
# artifact, not the fd bug, so the harness avoids it by default.
with SHARED_LOCK:
lpid = SHARED["lpid"]
if lpid and len(lpid) > 64:
h = lpid[:64]
flip = "0" if h[0] != "0" else "1"
return flip + h[1:] + lpid[64:]
return ("0" * 64) + "0"
def client_longpoll(cid):
while not STOP.is_set():
try:
s = mkconn(timeout=10.0)
except OSError:
ST.inc("connfail"); time.sleep(0.2); continue
rd = Reader(s)
params = [{"longpollid": _stale_lpid()}]
rid = next_id()
try:
s.sendall(req_bytes("getblocktemplate", params, rid, keepalive=False))
ST.inc("sent")
if random.random() < 0.5:
# impatient pool worker: abandon the request after 1-500 ms
time.sleep(random.uniform(0.001, 0.5))
end_early(s); continue
st, hd, bd = rd.read()
j = account(st, bd, rid, "getblocktemplate(lp)")
if j and isinstance(j, dict) and j.get("result"):
with SHARED_LOCK:
SHARED["lpid"] = j["result"].get("longpollid")
except (OSError, socket.timeout, EOFError, ValueError):
ST.inc("srvreset")
finally:
end_fin(s)
time.sleep(random.uniform(0.02, 0.2))
def miner(interval):
# Mine a block every `interval` seconds. Retry through work-queue rejections so
# blocks still land under heavy load (the production box mines steadily too).
while not STOP.is_set():
got = False
deadline = time.monotonic() + interval + 30
while not got and not STOP.is_set() and time.monotonic() < deadline:
try:
s = mkconn(timeout=60.0)
j, _ = call(s, Reader(s), "generate", [1], keepalive=False)
end_fin(s)
if j and not j.get("error") and j.get("result"):
ST.inc("mined"); got = True
h = j["result"][0] if isinstance(j["result"], list) else None
if h:
with SHARED_LOCK:
SHARED["hashes"].append(h)
SHARED["hashes"] = SHARED["hashes"][-200:]
else:
time.sleep(0.2) # likely work-queue 500; back off and retry
except Exception:
time.sleep(0.2)
for _ in range(int(interval * 10)):
if STOP.is_set():
break
time.sleep(0.1)
# ------------------------------------------------------------------ bootstrap shared state
def bootstrap():
try:
s = mkconn(timeout=30.0); rd = Reader(s)
j, _ = call(s, rd, "getbestblockhash", [], keepalive=True)
if j and j.get("result"):
with SHARED_LOCK:
SHARED["hashes"].append(j["result"])
j, _ = call(s, rd, "getblockcount", [], keepalive=True)
cnt = j.get("result") if j else None
if isinstance(cnt, int):
for hh in range(max(0, cnt - 30), cnt + 1):
jj, _ = call(s, rd, "getblockhash", [hh], keepalive=True)
if jj and jj.get("result"):
with SHARED_LOCK:
SHARED["hashes"].append(jj["result"])
j, _ = call(s, rd, "listaddresses", [], keepalive=True)
j, _ = call(s, rd, "z_listaddresses", [], keepalive=True)
if j and isinstance(j.get("result"), list):
with SHARED_LOCK:
SHARED["zaddrs"] = j["result"][:16]
j, _ = call(s, rd, "getblocktemplate", [{}], keepalive=True)
if j and isinstance(j.get("result"), dict):
with SHARED_LOCK:
SHARED["lpid"] = j["result"].get("longpollid")
end_fin(s)
log("bootstrap: %d hashes, %d zaddrs, lpid=%s" %
(len(SHARED["hashes"]), len(SHARED["zaddrs"]), bool(SHARED["lpid"])))
except Exception as e:
log("bootstrap failed: %s" % e)
# ------------------------------------------------------------------ fd accounting
def count_fds(pid):
if not pid:
return (None, None, None)
base = "/proc/%d/fd" % pid
try:
fds = os.listdir(base)
except OSError:
return (None, None, None)
sock = files = 0
for fd in fds:
try:
tgt = os.readlink(os.path.join(base, fd))
except OSError:
continue
if tgt.startswith("socket:"):
sock += 1
elif tgt.startswith("/"):
files += 1
return (len(fds), sock, files)
def pid_alive(pid):
if not pid:
return True
try:
os.kill(pid, 0); return True
except OSError:
return False
# ------------------------------------------------------------------ main
def reporter(pid):
t0 = time.monotonic()
while not STOP.is_set():
for _ in range(10):
if STOP.is_set():
break
time.sleep(0.1)
w = ST.swap()
tot, sk, fl = count_fds(pid)
fdstr = ""
if tot is not None:
fdstr = " | fds=%d sock=%d file=%d" % (tot, sk, fl)
log("t=%4ds ok=%d rpcerr=%d wq500=%d 4xx=%d other=%d | "
"earlyRST=%d earlyFIN=%d srvRST=%d srvEOF=%d ctmo=%d connfail=%d | "
"idmis=%d malf=%d | mined=%d halfopen=%d%s" % (
int(time.monotonic() - t0),
w["ok"], w["rpcerr"], w["wq500"], w["http4xx"], w["httpother"],
w["early_rst"], w["early_fin"], w["srvreset"], w["srveof"],
w["ctimeout"], w["connfail"], w["idmismatch"], w["malformed"],
w["mined"], w["half_open"], fdstr))
if pid and not pid_alive(pid):
log("!!! NODE PROCESS %d IS GONE -- stopping harness" % pid)
STOP.set()
break
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, default=19461)
ap.add_argument("--user", default="ab")
ap.add_argument("--password", default="abpass")
ap.add_argument("--duration", type=int, default=300)
ap.add_argument("--ctimeout", type=float, default=15.0)
ap.add_argument("--normal", type=int, default=24)
ap.add_argument("--impatient", type=int, default=10)
ap.add_argument("--slowfire", type=int, default=3,
help="paced clients that fire a genuinely-slow call then disconnect")
ap.add_argument("--slowfire-pace", type=float, default=2.0)
ap.add_argument("--halfopen", type=int, default=6)
ap.add_argument("--pipeline", type=int, default=6)
ap.add_argument("--garbage", type=int, default=4)
ap.add_argument("--slowread", type=int, default=2)
ap.add_argument("--longpoll", type=int, default=6)
# burst-size must exceed (rpcthreads + rpcworkqueue) to overflow the queue even
# when calls are fast; with -rpcthreads=8 -rpcworkqueue=4 that threshold is 12.
ap.add_argument("--burst-threads", type=int, default=1)
ap.add_argument("--burst-size", type=int, default=24)
ap.add_argument("--burst-every", type=float, default=1.5)
ap.add_argument("--mine", action="store_true", help="run a miner thread")
ap.add_argument("--mine-interval", type=float, default=4.0)
ap.add_argument("--pid", type=int, default=0, help="node pid, for /proc fd + liveness")
ap.add_argument("--logfile", default="")
args = ap.parse_args()
CFG.host = args.host; CFG.port = args.port
CFG.auth = auth_header(args.user, args.password); CFG.ctimeout = args.ctimeout
if args.logfile:
LOGF[0] = open(args.logfile, "a")
log("hammer start host=%s port=%d duration=%ds pid=%s" %
(CFG.host, CFG.port, args.duration, args.pid or "-"))
bootstrap()
threads = []
def spawn(fn, n, *a):
for i in range(n):
t = threading.Thread(target=fn, args=(i,) + a, daemon=True)
t.start(); threads.append(t)
spawn(client_normal, args.normal)
spawn(client_impatient, args.impatient)
spawn(client_slowfire, args.slowfire, args.slowfire_pace)
spawn(client_half_open, args.halfopen)
spawn(client_pipeline, args.pipeline)
spawn(client_garbage, args.garbage)
spawn(client_slowread, args.slowread)
spawn(client_longpoll, args.longpoll)
for i in range(args.burst_threads):
t = threading.Thread(target=client_burst, args=(i, args.burst_size, args.burst_every), daemon=True)
t.start(); threads.append(t)
if args.mine:
t = threading.Thread(target=miner, args=(args.mine_interval,), daemon=True)
t.start(); threads.append(t)
rt = threading.Thread(target=reporter, args=(args.pid,), daemon=True)
rt.start()
end = time.monotonic() + args.duration
try:
while time.monotonic() < end and not STOP.is_set():
time.sleep(0.2)
except KeyboardInterrupt:
pass
STOP.set()
time.sleep(1.0)
log("=== TOTALS ===")
for k in KEYS:
if ST.total[k]:
log(" %-14s %d" % (k, ST.total[k]))
log("anomalies: %d" % len(ST.anom))
for t, kind, detail in ST.anom[:40]:
log(" %s %s %s" % (time.strftime("%H:%M:%S", time.localtime(t)), kind, detail))
if args.pid:
log("node alive at end: %s" % pid_alive(args.pid))
if __name__ == "__main__":
main()
qa/fd-guard/repro/tools/rpcchurn.sh
Executable
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
# RPC-driven P2P peer churn on the target (same mix as fdbug/p2p/rpcchurn.sh):
# disconnectnode / addnode onetry / setban+clearbanned / kill -9 + restart of helper2.
# usage: rpcchurn.sh <run dir> <seconds>
source "$REPRO/common.sh"
OUT=$1; DUR=${2:-300}
DD_T=$OUT/target; DD_H1=$OUT/helper1; DD_H2=$OUT/helper2
end=$(( $(date +%s) + DUR )); i=0; restarts=0
while [ "$(date +%s)" -lt "$end" ] && [ ! -e "$OUT/STOP" ]; do
i=$((i+1))
case $(( i % 5 )) in
0) clin "$DD_T" "$RPC_T" disconnectnode "127.0.0.1:$P2P_H1" >/dev/null 2>&1 ;;
1) clin "$DD_T" "$RPC_T" addnode "127.0.0.1:$P2P_H1" onetry >/dev/null 2>&1 ;;
2) clin "$DD_T" "$RPC_T" setban "127.0.0.1" add 1 >/dev/null 2>&1; sleep 0.3
clin "$DD_T" "$RPC_T" clearbanned >/dev/null 2>&1
clin "$DD_H2" "$RPC_H2" addnode "127.0.0.1:$P2P_T" onetry >/dev/null 2>&1 ;;
3) pid=$(cat "$DD_H2/regtest/hushd.pid" 2>/dev/null)
if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then
kill -9 "$pid" 2>/dev/null
for _ in $(seq 1 20); do kill -0 "$pid" 2>/dev/null || break; sleep 0.5; done
start_helper "$DD_H2" "$P2P_H2" "$RPC_H2" "$OUT/logs/helper2.txt"
waitrpc "$DD_H2" "$RPC_H2" 120 >/dev/null 2>&1
restarts=$((restarts+1))
fi ;;
4) clin "$DD_T" "$RPC_T" addnode "127.0.0.1:$P2P_H2" onetry >/dev/null 2>&1 ;;
esac
sleep 0.7
done
echo "rpcchurn: $i cycles, $restarts helper2 restarts"
qa/fd-guard/repro/tools/scan_bdb.py
Executable
+113
View File
@@ -0,0 +1,113 @@
#!/usr/bin/env python3
"""
scan_bdb.py -- find Berkeley DB pages written into files that must not contain them
(blocks/blk*.dat, blocks/rev*.dat). Stand-in for the production incident tool.
BDB writes page P of a database at file offset P*pagesize through pwrite(fd, ...). If
fd has been closed and its number reused by a block file, the page lands in the block
file at exactly P*pagesize. So at every pagesize-aligned offset we decode the PAGE
header (db_page.h, all little-endian on x86):
0 u32 lsn.file 4 u32 lsn.offset 8 u32 pgno 12 u32 prev_pgno
16 u32 next_pgno 20 u16 entries 22 u16 hf_offset
24 u8 level 25 u8 type
and a metadata page (pgno 0 / any meta type) also carries magic at 12 and pagesize at 20.
A hit needs ALL of:
* pgno == offset // pagesize (the self-locating property)
* type is a real BDB page type (1..16 except 0)
* level consistent with type (overflow/meta/hash: 0; btree leaf: 1; internal: 2..32)
* hf_offset <= pagesize; entries <= pagesize/2
* for overflow (7): entries (refcount) >= 1 and 0 < hf_offset (data length) <= pagesize-26
Strength: "STRONG" if lsn == (0,1) (pages rewritten by lsn_reset -- the production
signature) or a meta page with a valid magic, else "weak".
Also reports any aligned BDB meta magic (btree 0x053162, hash 0x061561, queue 0x042253,
heap 0x074582) even when pgno does not match.
usage: scan_bdb.py [--pagesize 4096] [--all-offsets] files...
exit 0 = clean, 2 = hits.
"""
import argparse, struct, sys
TYPES = {1: "duplicate", 2: "hash_unsorted", 3: "ibtree", 4: "irecno", 5: "lbtree", 6: "lrecno",
7: "overflow", 8: "hashmeta", 9: "btreemeta", 10: "qammeta", 11: "qamdata", 12: "ldup",
13: "hash", 14: "heapmeta", 15: "heap", 16: "iheap"}
MAGIC = {0x053162: "btree", 0x061561: "hash", 0x042253: "queue", 0x074582: "heap"}
META = {8, 9, 10, 14}
def level_ok(t, lvl):
if t in (5, 6, 12): # leaf pages
return lvl == 1
if t in (3, 4): # internal pages
return 2 <= lvl <= 32
return lvl == 0 # overflow / meta / hash / queue / heap
def check(buf, off, ps):
if len(buf) < 26:
return None
lf, lo, pgno, prv, nxt, ent, hfo, lvl, typ = struct.unpack_from("<IIIIIHHBB", buf, 0)
if typ not in TYPES or pgno != off // ps or not level_ok(typ, lvl):
return None
if typ in META:
magic, ver, psz = struct.unpack_from("<III", buf, 12)
if magic not in MAGIC:
return None
return ("STRONG", f"{TYPES[typ]} magic={MAGIC[magic]} ver={ver} pagesize={psz} lsn=({lf},{lo})")
if hfo > ps or ent > ps // 2:
return None
if typ == 7 and not (ent >= 1 and 0 < hfo <= ps - 26):
return None
strength = "STRONG" if (lf, lo) == (0, 1) else "weak"
return (strength, f"{TYPES[typ]} lsn=({lf},{lo}) prev={prv} next={nxt} entries={ent} hf_offset={hfo} level={lvl}")
def scan(path, ps):
hits, magic_only = [], []
with open(path, "rb") as f:
off = 0
while True:
buf = f.read(ps)
if not buf:
break
r = check(buf, off, ps)
if r:
hits.append((off, off // ps) + r)
elif len(buf) >= 16:
m = struct.unpack_from("<I", buf, 12)[0]
if m in MAGIC and buf[25] in META:
magic_only.append((off, MAGIC[m]))
off += len(buf)
return hits, magic_only, off
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--pagesize", type=int, default=4096)
ap.add_argument("--quiet", action="store_true")
ap.add_argument("files", nargs="+")
a = ap.parse_args()
total = strong = 0
for p in a.files:
try:
hits, mo, size = scan(p, a.pagesize)
except OSError as e:
print(f"{p}: ERROR {e}")
continue
s = sum(1 for h in hits if h[2] == "STRONG")
total += len(hits); strong += s
if hits or mo:
print(f"{p}: {len(hits)} BDB page(s) ({s} strong), {len(mo)} stray meta magic, size={size}")
if not a.quiet:
for h in hits[:20]:
print(f" off={h[0]} pgno={h[1]} {h[2]} {h[3]}")
for m in mo[:5]:
print(f" meta-magic off={m[0]} {m[1]}")
else:
print(f"{p}: clean ({size} bytes)")
print(f"TOTAL BDB pages found: {total} ({strong} strong)")
return 2 if total else 0
if __name__ == "__main__":
sys.exit(main())
qa/fd-guard/repro/tools/tally.py
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""tally.py <fdguard.log> -- attribute every fdguard report to (type, closer function, target class)
and, for EBADF/NON-OWNER reports, to the code that made the previous close of that fd number.
The point: show that the stray closes ORIGINATE in CNode code (net.cpp CloseSocketDisconnect /
~CNode closing a number they already closed) and that libevent / BDB / CreateNodeFromAcceptedSocket /
GetTransaction only appear as victims-turned-amplifiers downstream of that.
"""
import collections, re, sys
ORIGIN = ("_ZN5CNode21CloseSocketDisconnectEv", "_ZN5CNodeD1Ev", "_ZN5CNodeD2Ev", "_ZN11CNetCleanupD2Ev")
NAMES = {"_ZN5CNode21CloseSocketDisconnectEv": "CNode::CloseSocketDisconnect (net.cpp:572)",
"_ZN5CNodeD1Ev": "CNode::~CNode (net.cpp:2819)",
"_ZN11CNetCleanupD2Ev": "CNetCleanup::~CNetCleanup (net.cpp:2465, shutdown)",
"evhttp_connection_free": "libevent evhttp_connection_free (http.c:1244)",
"__os_closehandle": "BDB __os_closehandle",
"_Z28CreateNodeFromAcceptedSocketjbRK8CAddressS1_": "CreateNodeFromAcceptedSocket (net.cpp:1336)",
"_Z14GetTransactionRK7uint256R12CTransactionRS_b": "GetTransaction fclose (main.cpp)",
"epoll_nochangelist_add": "libevent epoll ADD", "epoll_nochangelist_del": "libevent epoll DEL/MOD"}
def fn(frame_line):
m = re.search(r"\+0x[0-9a-f]+ (\S+?)\+0x", frame_line)
return m.group(1) if m else "?"
def main(path):
reps, cur = [], None
for line in open(path, errors="replace"):
if line.startswith("=== FDGUARD #"):
cur = {"hdr": line.strip(), "lines": []}
reps.append(cur)
elif cur is not None:
if line.startswith("=== END"):
cur = None
else:
cur["lines"].append(line.rstrip("\n"))
tally = collections.Counter()
prev_tally = collections.Counter()
db_blk = []
for r in reps:
typ = r["hdr"].split()[3]
L = r["lines"]
cls = next((re.search(r"class=(\S+)", l).group(1) for l in L if " target=" in l and "class=" in l), "?")
try:
i = next(k for k, l in enumerate(L) if l.strip() == "closer stack:")
closer = fn(L[i + 1])
except StopIteration:
closer = "?"
prev = "-"
for k, l in enumerate(L):
if "previous successful close of this fd number" in l:
# first frame line after the target= line
for l2 in L[k + 1:k + 4]:
if l2.strip().startswith("#0"):
prev = fn(l2)
break
break
tally[(typ, NAMES.get(closer, closer), cls)] += 1
if typ in ("EBADF-CLOSE", "NON-OWNER-CLOSE"):
prev_tally[NAMES.get(prev, prev)] += 1
if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk"):
tgt = next((l.split("target=")[1].split()[0] for l in L if " target=" in l), "?")
db_blk.append((r["hdr"].split()[2], r["hdr"].split()[4], NAMES.get(closer, closer), tgt.rsplit("/", 2)[-2:]))
print(f"{len(reps)} reports in {path}")
print("\n count type class closer")
for (typ, closer, cls), n in tally.most_common():
print(f"{n:6d} {typ:17s} {cls:6s} {closer}")
print("\nEBADF + NON-OWNER reports by the code that made the PREVIOUS close of the same number:")
for k, n in prev_tally.most_common():
print(f"{n:6d} {k}")
o = sum(n for (typ, c, cls), n in tally.items() if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk") and c.startswith(("CNode", "CNetCleanup")))
a = sum(n for (typ, c, cls), n in tally.items() if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk")) - o
print(f"\nNON-OWNER closes of wallet/BDB-log/block files: {o + a} (by CNode code directly: {o}; by an amplifier: {a})")
for d in db_blk[:40]:
print(" ", d[0], d[1], "closer=" + d[2], "target=" + "/".join(d[3]))
if __name__ == "__main__":
main(sys.argv[1])
qa/fd-guard/repro/tools/tcpchurn.py
Executable
+147
View File
@@ -0,0 +1,147 @@
#!/usr/bin/env python3
"""
Raw TCP / TLS client churn against the target node's P2P port.
dragonxd P2P is TLS 1.3 only (wolfSSL), self-signed, no cert verification. The
accept path is net.cpp CreateNodeFromAcceptedSocket -> tlsmanager.accept()
(wolfSSL_accept), and teardown is CloseSocketDisconnect / ~CNode
(wolfSSL_shutdown + wolfSSL_free + CloseSocket). This driver pokes the accept and
TLS-teardown/error paths (SSL_read/SSL_write at net.cpp ~1005 and tlsmanager.cpp
~660-680) by opening many short-lived connections that each misbehave:
rst : connect, then abort with RST (SO_LINGER 0) before any TLS bytes
fin : connect, then close cleanly (FIN) before any TLS bytes
garbage : connect, send non-ClientHello bytes, then drop
stall : connect, send a few bytes, hold mid-handshake, drop
tlsjunk : complete a real TLS 1.3 handshake (verify off), send junk, drop
tlshalf : start TLS, send a partial ClientHello record, drop
Loopback only, against a port this harness started. --duration seconds.
"""
import argparse, random, socket, ssl, struct, sys, time, os
def mk():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(3.0)
return s
def rst_close(s):
try:
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
except Exception:
pass
try: s.close()
except Exception: pass
def do_rst(host, port):
s = mk()
try:
s.connect((host, port)); rst_close(s)
except Exception:
rst_close(s)
def do_fin(host, port):
s = mk()
try:
s.connect((host, port)); s.close()
except Exception:
try: s.close()
except Exception: pass
def do_garbage(host, port):
s = mk()
try:
s.connect((host, port))
s.sendall(os.urandom(random.randint(16, 400)))
time.sleep(random.uniform(0.0, 0.3))
except Exception:
pass
rst_close(s)
def do_stall(host, port):
s = mk()
try:
s.connect((host, port))
# a couple of bytes that look like the start of a TLS record, then wait
s.sendall(b"\x16\x03\x01")
time.sleep(random.uniform(0.5, 2.0))
except Exception:
pass
rst_close(s)
def _ctx():
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
try:
ctx.minimum_version = ssl.TLSVersion.TLSv1_3
except Exception:
pass
return ctx
def do_tlsjunk(host, port):
raw = mk()
try:
raw.connect((host, port))
tls = _ctx().wrap_socket(raw, server_hostname=None)
# handshake completed; the node now expects a P2P version message.
# send junk inside the TLS channel, then tear down.
try:
tls.sendall(os.urandom(random.randint(8, 128)))
except Exception:
pass
time.sleep(random.uniform(0.0, 0.4))
try:
tls.close()
except Exception:
rst_close(raw)
except Exception:
rst_close(raw)
def do_tlshalf(host, port):
s = mk()
try:
s.connect((host, port))
# a well-formed TLS record header claiming a long handshake, but we send
# only part of it, then drop -> server sits in wolfSSL_accept WANT_READ.
body = os.urandom(40)
rec = b"\x16\x03\x03" + struct.pack(">H", 512) + body
s.sendall(rec)
time.sleep(random.uniform(0.3, 1.5))
except Exception:
pass
rst_close(s)
HANDLERS = [do_rst, do_fin, do_garbage, do_stall, do_tlsjunk, do_tlshalf]
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, required=True)
ap.add_argument("--duration", type=float, default=300.0)
ap.add_argument("--concurrency", type=int, default=6)
ap.add_argument("--rate", type=float, default=0.03, help="sleep between bursts")
args = ap.parse_args()
end = time.time() + args.duration
counts = {h.__name__: 0 for h in HANDLERS}
import threading
lock = threading.Lock()
def worker():
while time.time() < end:
h = random.choice(HANDLERS)
h(args.host, args.port)
with lock:
counts[h.__name__] += 1
time.sleep(random.uniform(0.0, args.rate))
ts = [threading.Thread(target=worker, daemon=True) for _ in range(args.concurrency)]
for t in ts: t.start()
for t in ts: t.join()
total = sum(counts.values())
print(f"tcpchurn: {total} connections in {args.duration:.0f}s -> {counts}", flush=True)
if __name__ == "__main__":
main()
qa/fd-guard/shim/build.sh
Executable
+15
View File
@@ -0,0 +1,15 @@
#!/bin/sh
# Build the fdguard LD_PRELOAD shim and its self-test program.
# ./build.sh -> libfdguard.so, fdguard_test
set -e
cd "$(dirname "$0")"
CC=${CC:-gcc}
$CC -O2 -g -fPIC -shared -Wall -Wextra -Wno-unused-parameter -fvisibility=default \
-o libfdguard.so.tmp fdguard.c -ldl -lpthread
# rename, never overwrite in place: a running process may have the old one mapped
mv -f libfdguard.so.tmp libfdguard.so
# the test program keeps its symbols (no -rdynamic on purpose: like dragonxd,
# owner detection must work from .symtab alone) and frame pointers for clarity
$CC -O1 -g -fno-omit-frame-pointer -fno-inline -Wall -o fdguard_test.tmp fdguard_test.c -lpthread
mv -f fdguard_test.tmp fdguard_test
echo "built: $(pwd)/libfdguard.so $(pwd)/fdguard_test"
qa/fd-guard/shim/fdguard.c
+1500
View File
File diff suppressed because it is too large Load Diff
qa/fd-guard/shim/fdguard_test.c
+206
View File
@@ -0,0 +1,206 @@
/*
* fdguard_test.c -- exercises libfdguard.so.
*
* The function names imitate the real owners inside dragonxd so the shim's
* built-in domain patterns apply unchanged:
* __os_openhandle / __os_closehandle Berkeley DB os layer (domain bdb)
* OpenDiskFile / FlushBlockFile block-file code (domain blk)
* ConnectSocketDirectly / CloseSocket P2P sockets (domain p2p)
* foreign_stale_close / foreign_* somebody who holds a stale number
*
* usage: fdguard_test <dir> <case> (see main() for the list)
*/
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/epoll.h>
#include <sys/socket.h>
#include <sys/syscall.h>
#include <time.h>
#include <unistd.h>
#define NI __attribute__((noinline))
static char D[512];
static char *P(const char *rel) { static char b[4][700]; static int i; i = (i + 1) & 3; snprintf(b[i], sizeof b[i], "%s/%s", D, rel); return b[i]; }
static void say(const char *m) { fprintf(stderr, "[test] %s\n", m); }
/* --- "Berkeley DB" --- */
NI int __os_openhandle(const char *path) { return open(path, O_RDWR | O_CREAT, 0600); }
NI int __os_closehandle(int fd) { return close(fd); }
/* --- "block files" --- */
NI FILE *OpenDiskFile(const char *path) { FILE *f = fopen(path, "rb+"); if (!f) f = fopen(path, "wb+"); return f; }
NI int FlushBlockFile(FILE *f) { fflush(f); return fclose(f); }
/* --- "P2P" --- */
NI int ConnectSocketDirectly(void) { return socket(AF_INET, SOCK_STREAM, 0); }
NI int CloseSocket(int *s) { int r = close(*s); if (r) fprintf(stderr, "[test] Socket close failed: %d. Error: %s\n", *s, strerror(errno)); return r; }
/* --- foreigners --- */
NI int foreign_stale_close(int fd) { return close(fd); }
NI int foreign_stale_fclose(FILE *f) { return fclose(f); }
NI int foreign_dup2(int fd) { int dn = open("/dev/null", O_RDWR); int r = dup2(dn, fd); close(dn); return r; }
NI int foreign_close_range(int fd) { return close_range((unsigned)fd, (unsigned)fd, 0); }
NI void foreign_closefrom(int fd) { closefrom(fd); }
/* --- plain code that opens and closes in the same function --- */
NI int copy_like(const char *src) { int fd = open(src, O_RDONLY); char b[16]; if (fd >= 0) { (void)!read(fd, b, sizeof b); close(fd); } return fd; }
NI int plain_open(const char *p) { return open(p, O_RDWR | O_CREAT, 0600); }
NI int plain_close(int fd) { return close(fd); }
static int fd_valid(int fd) { return fcntl(fd, F_GETFD) != -1; }
static int c_double(void)
{
int fd = plain_open(P("plain.txt"));
plain_close(fd);
int r = plain_close(fd); /* EBADF */
return r == -1 && errno == EBADF ? 0 : 1;
}
static int c_dbowner(void) { int fd = __os_openhandle(P("regtest/wallet.dat")); int r = __os_closehandle(fd); return r; }
static int c_dbforeign(void)
{
int fd = __os_openhandle(P("regtest/wallet.dat"));
foreign_stale_close(fd); /* HIT */
int r = __os_closehandle(fd); /* BDB's own close then fails: EBADF (unless deny) */
(void)r;
return 0;
}
static int c_deny(void)
{
int fd = __os_openhandle(P("regtest/wallet.dat"));
foreign_stale_close(fd); /* HIT + DENIED */
if (!fd_valid(fd)) { say("deny FAILED: fd closed"); return 1; }
if (write(fd, "x", 1) != 1) { say("deny FAILED: write"); return 1; }
return __os_closehandle(fd);
}
static int c_blkowner(void) { FILE *f = OpenDiskFile(P("regtest/blocks/blk00000.dat")); fputs("blk", f); return FlushBlockFile(f); }
static int c_blkraw(void)
{
FILE *f = OpenDiskFile(P("regtest/blocks/blk00001.dat"));
foreign_stale_close(fileno(f)); /* HIT: raw close under a live FILE */
FlushBlockFile(f); /* EBADF from fclose */
return 0;
}
static int c_stalefile(void)
{
FILE *f = OpenDiskFile(P("regtest/blocks/blk00002.dat"));
int n = fileno(f);
foreign_stale_close(n); /* HIT #1 */
int w = __os_openhandle(P("regtest/wallet.dat"));
if (w != n) fprintf(stderr, "[test] note: wallet got fd %d, FILE had %d\n", w, n);
FlushBlockFile(f); /* HIT #2: fclose of a stale FILE whose number is now wallet.dat */
if (w == n) __os_closehandle(w); /* EBADF: BDB lost its fd */
return 0;
}
static int c_reuse(void)
{
int s = ConnectSocketDirectly();
int keep = s;
CloseSocket(&s); /* legit */
int w = __os_openhandle(P("regtest/wallet.dat")); /* reuses the number */
if (w != keep) fprintf(stderr, "[test] note: wallet got fd %d, socket was %d\n", w, keep);
s = keep;
CloseSocket(&s); /* HIT: stale socket number now = wallet.dat */
__os_closehandle(w); /* EBADF */
return 0;
}
static int c_dup2(void) { int w = __os_openhandle(P("regtest/wallet.dat")); foreign_dup2(w); __os_closehandle(w); return 0; }
static int c_range(void)
{
int w = __os_openhandle(P("regtest/wallet.dat"));
foreign_close_range(w); /* HIT */
w = __os_openhandle(P("regtest/database/log.0000000001"));
int hi = fcntl(w, F_DUPFD, 600); /* move it high so closefrom only hits it */
close(w);
foreign_closefrom(hi); /* HIT (domain mismatch: opened by test main via fcntl dup? -> fallback db) */
return 0;
}
static int c_owner_misc(void)
{
/* nothing here may be flagged */
copy_like(P("regtest/wallet.dat")); /* common-caller */
int fd = __os_openhandle(P("regtest/blocks/blk00003.dat"));
FILE *f = fdopen(fd, "rb+"); /* ownership moves to FILE */
FlushBlockFile(f);
int s = ConnectSocketDirectly(); CloseSocket(&s);
fd = (int)syscall(SYS_openat, AT_FDCWD, P("regtest/wallet.dat"), O_RDWR); /* untracked open */
__os_closehandle(fd); /* fallback:bdb */
return 0;
}
static int c_fallback_hit(void)
{
int fd = (int)syscall(SYS_openat, AT_FDCWD, P("regtest/wallet.dat"), O_RDWR); /* untracked open */
foreign_stale_close(fd); /* HIT via fallback rule */
return 0;
}
static int c_epoll(void)
{
int ep = epoll_create1(0);
int fd = plain_open(P("plain2.txt"));
struct epoll_event ev = { .events = EPOLLIN };
int r = epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev); /* EPERM: regular file */
close(fd);
r = epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev); /* EBADF */
close(ep);
return r == -1 ? 0 : 1;
}
static int c_selfclose(void)
{
/* close every fd from 3 to 1100 by number, like a buggy daemonizer */
for (int i = 3; i < 1100; i++) close(i);
return 0;
}
/* concurrency + micro-benchmark: open/close churn in many threads */
static int nthr = 8, iters = 20000;
static void *worker(void *arg)
{
long id = (long)arg;
char p[700]; snprintf(p, sizeof p, "%s/thr%ld.txt", D, id);
for (int i = 0; i < iters; i++) {
int s = ConnectSocketDirectly(); CloseSocket(&s);
int fd = plain_open(p); plain_close(fd);
FILE *f = fopen(p, "r"); if (f) fclose(f);
if ((i & 255) == 0) { int w = __os_openhandle(P("regtest/wallet.dat")); __os_closehandle(w); }
}
return NULL;
}
static int c_threads(void)
{
const char *e = getenv("T_ITERS"); if (e) iters = atoi(e);
pthread_t t[64];
struct timespec a, b; clock_gettime(CLOCK_MONOTONIC, &a);
for (long i = 0; i < nthr; i++) pthread_create(&t[i], NULL, worker, (void *)i);
for (int i = 0; i < nthr; i++) pthread_join(t[i], NULL);
clock_gettime(CLOCK_MONOTONIC, &b);
double ns = (b.tv_sec - a.tv_sec) * 1e9 + (b.tv_nsec - a.tv_nsec);
long ops = (long)nthr * iters * 3;
printf("threads=%d iters=%d ops=%ld (socket/open/fopen + close each) wall=%.3fs %.0f ns/op-pair/thread\n",
nthr, iters, ops, ns / 1e9, ns * nthr / ops);
return 0;
}
int main(int argc, char **argv)
{
if (argc < 3) { fprintf(stderr, "usage: %s <dir> <case>\n", argv[0]); return 2; }
snprintf(D, sizeof D, "%s", argv[1]);
const char *c = argv[2];
if (!strcmp(c, "double")) return c_double();
if (!strcmp(c, "dbowner")) return c_dbowner();
if (!strcmp(c, "dbforeign")) return c_dbforeign();
if (!strcmp(c, "deny")) return c_deny();
if (!strcmp(c, "blkowner")) return c_blkowner();
if (!strcmp(c, "blkraw")) return c_blkraw();
if (!strcmp(c, "stalefile")) return c_stalefile();
if (!strcmp(c, "reuse")) return c_reuse();
if (!strcmp(c, "dup2")) return c_dup2();
if (!strcmp(c, "range")) return c_range();
if (!strcmp(c, "ownermisc")) return c_owner_misc();
if (!strcmp(c, "fallback")) return c_fallback_hit();
if (!strcmp(c, "epoll")) return c_epoll();
if (!strcmp(c, "selfclose")) return c_selfclose();
if (!strcmp(c, "threads")) return c_threads();
fprintf(stderr, "unknown case %s\n", c);
return 2;
}
qa/fd-guard/shim/run_tests.sh
Executable
+97
View File
@@ -0,0 +1,97 @@
#!/bin/bash
# Self-test for libfdguard.so. Every case lists the report types it must
# produce (exactly, in order) -- an empty list means "no report at all".
# ./run_tests.sh (builds first)
set -u
cd "$(dirname "$0")"
./build.sh >/dev/null || { echo "build failed"; exit 1; }
SO=$PWD/libfdguard.so
T=$PWD/t
rm -rf "$T"; mkdir -p "$T/regtest/blocks" "$T/regtest/database"
pass=0; fail=0
run() { # run <case> [ENV=VAL ...] -> sets RC, LOG
local c=$1; shift
LOG=$T/$c${SUFFIX:-}.log; rm -f "$LOG"
env FDGUARD_EXE=fdguard_test FDGUARD_LOG="$LOG" "$@" LD_PRELOAD="$SO" ./fdguard_test "$T" "$c" >"$T/$c${SUFFIX:-}.out" 2>&1
RC=$?
}
types() { grep '^=== FDGUARD' "$LOG" 2>/dev/null | awk '{print $4}' | tr '\n' ' ' | sed 's/ $//'; }
reasons() { grep -o 'reason=[^ ]*' "$LOG" 2>/dev/null | sed 's/reason=//' | tr '\n' ' ' | sed 's/ $//'; }
check() { # check <name> <cond-description> <actual> <expected>
if [ "$3" == "$4" ]; then pass=$((pass+1)); printf 'PASS %-12s %s\n' "$1" "$2";
else fail=$((fail+1)); printf 'FAIL %-12s %s\n got: [%s]\n want: [%s]\n' "$1" "$2" "$3" "$4"; fi
}
# case | expected report types | expected reasons
while IFS='|' read -r c want wantr; do
[ -z "$c" ] && continue
run "$c"
check "$c" "reports" "$(types)" "$want"
check "$c" "reasons" "$(reasons)" "$wantr"
check "$c" "exit 0" "$RC" "0"
done <<'EOF'
double|EBADF-CLOSE|
dbowner||
dbforeign|NON-OWNER-CLOSE EBADF-CLOSE|DOMAIN-MISMATCH
blkowner||
blkraw|NON-OWNER-CLOSE EBADF-CLOSE|stdio:RAW-CLOSE-UNDER-LIVE-FILE
stalefile|NON-OWNER-CLOSE NON-OWNER-CLOSE EBADF-CLOSE|stdio:RAW-CLOSE-UNDER-LIVE-FILE stdio:STALE-FILE
reuse|NON-OWNER-CLOSE EBADF-CLOSE|DOMAIN-MISMATCH
dup2|NON-OWNER-CLOSE|DOMAIN-MISMATCH
range|NON-OWNER-CLOSE NON-OWNER-CLOSE NON-OWNER-CLOSE|DOMAIN-MISMATCH DOMAIN-MISMATCH fallback:DB-FILE-CLOSED-OUTSIDE-BDB
ownermisc||
fallback|NON-OWNER-CLOSE|fallback:DB-FILE-CLOSED-OUTSIDE-BDB
epoll|EPOLL-CTL-EPERM EPOLL-CTL-EBADF|epoll_ctl(ADD)-failed epoll_ctl(ADD)-failed
EOF
# the previous close of the reused number must be the first CloseSocket (proves the double close)
run reuse
check reuse "history shows 1st CloseSocket on a socket" \
"$(grep -A2 'previous successful close' "$LOG" | head -3 | grep -c -e 'class=sock dom=p2p' -e 'CloseSocket+')" "2"
# the stale FILE report must name wallet.dat as the victim and OpenDiskFile as the FILE's opener
run stalefile
check stalefile "victim=wallet.dat, FILE opened in OpenDiskFile" \
"$(sed -n '/#2 NON-OWNER/,/=== END #2/p' "$LOG" | grep -m1 -c 'target=.*wallet.dat class=db') $(sed -n '/#2 NON-OWNER/,/=== END #2/p' "$LOG" | grep -A3 'which had been opened by: fopen' | grep -c 'OpenDiskFile+')" "1 1"
# the log fd survives a close-everything loop and the attempt is reported
run selfclose
check selfclose "log-fd close refused + reported" "$(grep -c 'CLOSE-OF-FDGUARD-LOG-FD' "$LOG")" "1"
check selfclose "log still writable afterwards (final stats)" "$(grep -c '^S .* final' "$LOG")" "1"
# abort mode: first non-owner close -> SIGABRT (rc 134), report already written
SUFFIX=_abort run dbforeign FDGUARD_ABORT=1
check abort "rc=134 (SIGABRT)" "$RC" "134"
check abort "report written before abort" "$(types)" "NON-OWNER-CLOSE"
# abort=2 also aborts on EBADF
SUFFIX=_abort2 run double FDGUARD_ABORT=2
check abort2 "rc=134 on EBADF" "$RC" "134"
# deny mode: wallet fd survives the foreign close; no EBADF afterwards
SUFFIX=_deny run deny FDGUARD_DENY=1
check deny "exit 0 (fd still valid + writable)" "$RC" "0"
check deny "reported as DENIED, no EBADF" "$(types) $(grep -c 'ACTION=DENIED' "$LOG")" "NON-OWNER-CLOSE 1"
# extra classes: guard=all with the owner cases must still be silent
SUFFIX=_all run ownermisc FDGUARD_GUARD=all
check guard-all "ownermisc silent with FDGUARD_GUARD=all" "$(types)" ""
# trace-all writes one T line per open/close
SUFFIX=_trace run dbowner FDGUARD_TRACE_ALL=1
check trace "T lines for open+close" "$(grep -c '^T ' "$LOG")" "2"
# passive in non-matching processes
LOG=$T/passive.log; env FDGUARD_EXE=dragonxd FDGUARD_LOG="$LOG" LD_PRELOAD="$SO" ./fdguard_test "$T" dbforeign >/dev/null 2>&1
check passive "no log when exe does not match" "$( [ -e "$LOG" ] && echo exists || echo none)" "none"
# concurrency: 8 threads x 5000 x (socket, open, fopen) churn, guard=all -> no reports
SUFFIX=_thr run threads T_ITERS=5000 FDGUARD_GUARD=all
check threads "no reports under 8-thread churn (guard=all)" "$(types)" ""
check threads "stats: HITS=0 EBADF=0" "$(grep '^S .*final' "$LOG" | grep -o 'HITS=[0-9]* EBADF=[0-9]*')" "HITS=0 EBADF=0"
# symbolizer resolves file:line for the test binary
check symbolize "addr2line resolved CloseSocket to fdguard_test.c" \
"$(./symbolize.py "$T/reuse.log" | grep -m1 -o 'in CloseSocket at .*fdguard_test.c:[0-9]*' | sed 's/.*fdguard_test.c:[0-9]*/ok/')" "ok"
echo "== $pass passed, $fail failed"
[ $fail -eq 0 ]
qa/fd-guard/shim/symbolize.py
Executable
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env python3
"""
symbolize.py -- turn fdguard log frames into function + file:line.
Every stack frame in an fdguard report looks like
#3 0x5b96b433c991 /path/to/module+0x1991 c_dbforeign+0x2d [bdb]
where "+0x1991" is the ELF virtual address inside that module (runtime address
minus the module's load bias, so PIE executables and shared libraries need no
further adjustment; for a non-PIE executable it equals the absolute address).
Frames are return addresses, so we ask addr2line about vaddr-1 (the call
instruction), with -f -C -i (function, demangle, inline chain).
backtrace_symbols_fd lines "module(+0x1991)[0x...]" are symbolized too.
usage:
symbolize.py [options] LOG [LOG...] annotated log on stdout
symbolize.py --summary LOG one line per distinct report
options:
--map OLD=NEW read module OLD from path NEW instead (repeatable),
e.g. --map /path/to/deployed/dragonxd=./dragonxd.unstripped
--addr2line PATH addr2line binary (default: addr2line)
--only-reports drop T/S lines (trace + stats) from the output
"""
import os
import re
import subprocess
import sys
from collections import defaultdict, OrderedDict
FRAME_RE = re.compile(r'^(\s*#\d+ 0x[0-9a-f]+ )(\S+)\+0x([0-9a-f]+)(.*)$')
BSFD_RE = re.compile(r'^(\S+)\(\+0x([0-9a-f]+)\)\[0x[0-9a-f]+\]\s*$')
HDR_RE = re.compile(r'^=== FDGUARD #(\d+) (\S+)')
def parse_args(argv):
opts = {'map': {}, 'addr2line': 'addr2line', 'summary': False, 'only': False, 'logs': []}
it = iter(argv)
for a in it:
if a == '--map':
old, new = next(it).split('=', 1)
opts['map'][old] = new
elif a.startswith('--map='):
old, new = a[6:].split('=', 1)
opts['map'][old] = new
elif a == '--addr2line':
opts['addr2line'] = next(it)
elif a == '--summary':
opts['summary'] = True
elif a == '--only-reports':
opts['only'] = True
elif a in ('-h', '--help'):
print(__doc__)
sys.exit(0)
else:
opts['logs'].append(a)
if not opts['logs']:
opts['logs'] = ['-']
return opts
def read_lines(paths):
out = []
for p in paths:
f = sys.stdin if p == '-' else open(p, errors='replace')
out.extend(f.read().splitlines())
if f is not sys.stdin:
f.close()
return out
def resolve_module(mod, opts, known):
if mod in opts['map']:
return opts['map'][mod]
if os.path.exists(mod):
return mod
base = os.path.basename(mod)
for k in known:
if os.path.basename(k) == base and os.path.exists(opts['map'].get(k, k)):
return opts['map'].get(k, k)
return None
def run_addr2line(binary, addrs, a2l):
"""addrs: list of int vaddrs (already -1'd). returns {addr: [(func, loc), ...]}"""
res = {}
if not addrs:
return res
uniq = sorted(set(addrs))
try:
p = subprocess.run([a2l, '-a', '-f', '-C', '-i', '-e', binary] + ['0x%x' % a for a in uniq],
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, check=False)
except FileNotFoundError:
return res
cur = None
lines = p.stdout.splitlines()
i = 0
while i < len(lines):
ln = lines[i]
if re.match(r'^0x[0-9a-f]+$', ln):
cur = int(ln, 16)
res[cur] = []
i += 1
continue
func = ln
loc = lines[i + 1] if i + 1 < len(lines) else '??:0'
if cur is not None:
res[cur].append((func, loc))
i += 2
return res
def main():
opts = parse_args(sys.argv[1:])
lines = read_lines(opts['logs'])
if opts['summary']:
summary(lines)
return
# pass 1: collect modules + addresses
want = defaultdict(set)
known = OrderedDict()
for ln in lines:
m = FRAME_RE.match(ln)
if m:
known[m.group(2)] = 1
want[m.group(2)].add(max(int(m.group(3), 16) - 1, 0))
for ln in lines:
m = BSFD_RE.match(ln)
if m:
want[m.group(1)].add(max(int(m.group(2), 16) - 1, 0))
table = {}
for mod, addrs in want.items():
path = resolve_module(mod, opts, known)
if not path:
continue
table[mod] = run_addr2line(path, list(addrs), opts['addr2line'])
def fmt(mod, vaddr):
r = table.get(mod, {}).get(vaddr)
if not r:
return []
out = []
for j, (func, loc) in enumerate(r):
if func == '??' and loc.startswith('??'):
continue
# addr2line -i lists the innermost (inlined) frame first
tag = 'inlined:' if j < len(r) - 1 else 'in'
out.append('%s %s at %s' % (tag, func, loc))
return out
w = sys.stdout.write
for ln in lines:
if opts['only'] and (ln.startswith('T ') or ln.startswith('S ')):
continue
w(ln + '\n')
m = FRAME_RE.match(ln)
if m:
v = max(int(m.group(3), 16) - 1, 0)
for s in fmt(m.group(2), v):
w(' -> ' + s + '\n')
continue
m = BSFD_RE.match(ln)
if m:
v = max(int(m.group(2), 16) - 1, 0)
for s in fmt(m.group(1), v):
w(' -> ' + s + '\n')
def summary(lines):
"""one line per (type, reason, class, first exe closer frame)"""
groups = OrderedDict()
cur = None
in_closer = False
for ln in lines:
h = HDR_RE.match(ln)
if h:
cur = {'type': h.group(2), 'reason': '', 'cls': '', 'top': '', 'call': ''}
in_closer = False
continue
if cur is None:
continue
s = ln.strip()
if s.startswith('call:'):
cur['call'] = s.split('(')[0][6:]
elif s.startswith('target=') and ' class=' in s and not cur['cls']:
cur['cls'] = s.rsplit('class=', 1)[-1]
elif s.startswith('verdict='):
cur['reason'] = s.split('reason=', 1)[-1].split(' ')[0]
elif s == 'closer stack:':
in_closer = True
elif in_closer and FRAME_RE.match(ln):
parts = s.split(' ')
if len(parts) >= 4 and not cur['top'] and '/libc.so' not in parts[2] and '/libstdc++' not in parts[2]:
cur['top'] = parts[3]
elif in_closer and not s.startswith('#'):
in_closer = False
if s.startswith('=== END'):
k = (cur['type'], cur['call'], cur['reason'], cur['cls'], cur['top'])
groups[k] = groups.get(k, 0) + 1
cur = None
print('%6s %-22s %-12s %-6s %-40s %s' % ('count', 'type', 'call', 'class', 'reason', 'first closer frame'))
for (t, c, r, cl, top), n in groups.items():
print('%6d %-22s %-12s %-6s %-40s %s' % (n, t[:22], c, cl, r[:40], top))
stats = [ln for ln in lines if ln.startswith('S ')]
if stats:
print('\nlast stats line:\n ' + stats[-1])
if __name__ == '__main__':
main()
src/Makefile.gtest.include
+6 -2
View File
@@ -15,8 +15,12 @@ hush_gtest_SOURCES = \
gtest/test_hdtransparent.cpp \ gtest/test_hdtransparent.cpp \
gtest/test_mnemonic_compat.cpp \ gtest/test_mnemonic_compat.cpp \
gtest/test_stratum_jobid.cpp \ gtest/test_stratum_jobid.cpp \
gtest/test_wallet_scan_notes.cpp \ gtest/test_wallet_notedata.cpp \
gtest/test_operation_status.cpp gtest/test_net_closesocket.cpp \
gtest/test_fd_defense.cpp \
gtest/test_wallet_dbenv.cpp \
gtest/test_wallet_notecache.cpp \
gtest/test_wallet_history.cpp
hush_gtest_CPPFLAGS = $(AM_CPPFLAGS) -DMULTICORE -fopenmp -DBINARY_OUTPUT -DCURVE_ALT_BN128 -DSTATIC $(BITCOIN_INCLUDES) hush_gtest_CPPFLAGS = $(AM_CPPFLAGS) -DMULTICORE -fopenmp -DBINARY_OUTPUT -DCURVE_ALT_BN128 -DSTATIC $(BITCOIN_INCLUDES)
hush_gtest_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS) hush_gtest_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
src/asyncrpcoperation.cpp
+3 -3
View File
@@ -32,7 +32,7 @@ using namespace std;
static boost::uuids::random_generator uuidgen; static boost::uuids::random_generator uuidgen;
static const std::map<OperationStatus, std::string> OperationStatusMap = { static std::map<OperationStatus, std::string> OperationStatusMap = {
{OperationStatus::READY, "queued"}, {OperationStatus::READY, "queued"},
{OperationStatus::EXECUTING, "executing"}, {OperationStatus::EXECUTING, "executing"},
{OperationStatus::CANCELLED, "cancelled"}, {OperationStatus::CANCELLED, "cancelled"},
@@ -194,7 +194,7 @@ UniValue AsyncRPCOperation::getStatus() const {
OperationStatus status = this->getState(); OperationStatus status = this->getState();
UniValue obj(UniValue::VOBJ); UniValue obj(UniValue::VOBJ);
obj.push_back(Pair("id", this->id_)); obj.push_back(Pair("id", this->id_));
obj.push_back(Pair("status", OperationStatusMap.at(status))); obj.push_back(Pair("status", OperationStatusMap[status]));
obj.push_back(Pair("creation_time", this->creation_time_)); obj.push_back(Pair("creation_time", this->creation_time_));
// TODO: Issue #1354: There may be other useful metadata to return to the user. // TODO: Issue #1354: There may be other useful metadata to return to the user.
UniValue err = this->getError(); UniValue err = this->getError();
@@ -218,5 +218,5 @@ UniValue AsyncRPCOperation::getStatus() const {
*/ */
std::string AsyncRPCOperation::getStateAsString() const { std::string AsyncRPCOperation::getStateAsString() const {
OperationStatus status = this->getState(); OperationStatus status = this->getState();
return OperationStatusMap.at(status); return OperationStatusMap[status];
} }
src/asyncrpcqueue.cpp
+4 -8
View File
@@ -129,18 +129,14 @@ std::shared_ptr<AsyncRPCOperation> AsyncRPCQueue::getOperationForId(AsyncRPCOper
/** /**
* Return the operation for a given operation id and then remove the operation from internal storage. * Return the operation for a given operation id and then remove the operation from internal storage.
* The lookup and the erase happen under one hold of lock_, so when several threads pop the same id
* exactly one of them receives the operation and the others get a null pointer.
*/ */
std::shared_ptr<AsyncRPCOperation> AsyncRPCQueue::popOperationForId(AsyncRPCOperationId id) { std::shared_ptr<AsyncRPCOperation> AsyncRPCQueue::popOperationForId(AsyncRPCOperationId id) {
std::shared_ptr<AsyncRPCOperation> ptr; // declared before guard, so it is released after unlock std::shared_ptr<AsyncRPCOperation> ptr = getOperationForId(id);
std::lock_guard<std::mutex> guard(lock_); if (ptr) {
AsyncRPCOperationMap::iterator iter = operation_map_.find(id); std::lock_guard<std::mutex> guard(lock_);
if (iter != operation_map_.end()) {
ptr = iter->second;
// Note: if the id still exists in the operationIdQueue, when it gets processed by a worker // Note: if the id still exists in the operationIdQueue, when it gets processed by a worker
// there will no operation in the map to execute, so nothing will happen. // there will no operation in the map to execute, so nothing will happen.
operation_map_.erase(iter); operation_map_.erase(id);
} }
return ptr; return ptr;
} }
src/gtest/test_fd_defense.cpp
+83
View File
@@ -0,0 +1,83 @@
// Copyright (c) 2016-2024 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
#include <gtest/gtest.h>
#include <atomic>
#include <errno.h>
#include <stdexcept>
#include <stdio.h>
#include <unistd.h>
#include <boost/filesystem.hpp>
#include <boost/thread.hpp>
#include "chainparams.h"
#include "init.h"
#include "main.h"
#include "noui.h"
#include "util.h"
#include "validationinterface.h"
extern std::atomic<bool> fRequestShutdown;
// A block file opened for reading must not be writable through its fd: a stray write through a reused fd number
// (the wolfSSL CloseSocket macro let the wallet's BDB fd be closed behind its back) used to land in it.
TEST(FdDefense, ReadOnlyBlockFileIsNotWritable) {
SelectParams(CBaseChainParams::MAIN);
const boost::filesystem::path dir = boost::filesystem::temp_directory_path() / boost::filesystem::unique_path("fddefense-%%%%%%%%");
const bool hadDatadir = mapArgs.count("-datadir") > 0;
const std::string oldDatadir = hadDatadir ? mapArgs["-datadir"] : "";
boost::filesystem::create_directories(dir);
mapArgs["-datadir"] = dir.string();
ClearDatadirCache();
const CDiskBlockPos pos(0, 0);
const boost::filesystem::path blk = GetBlockPosFilename(pos, "blk");
boost::filesystem::create_directories(blk.parent_path());
{
FILE* f = fopen(blk.string().c_str(), "wb");
ASSERT_TRUE(f != nullptr);
fputs("block bytes", f);
fclose(f);
}
FILE* ro = OpenBlockFile(pos, true);
ASSERT_TRUE(ro != nullptr);
char c = 0;
EXPECT_EQ(1u, fread(&c, 1, 1, ro)) << "still readable";
errno = 0;
EXPECT_EQ(-1, pwrite(fileno(ro), "X", 1, 0)) << "a write through a read-only block file's fd must fail";
EXPECT_EQ(EBADF, errno);
fclose(ro);
FILE* rw = OpenBlockFile(pos, false);
ASSERT_TRUE(rw != nullptr);
EXPECT_EQ(1, pwrite(fileno(rw), "b", 1, 0)) << "the writable open still writes";
fclose(rw);
if (hadDatadir) mapArgs["-datadir"] = oldDatadir; else mapArgs.erase("-datadir");
ClearDatadirCache();
boost::filesystem::remove_all(dir);
}
// A wallet notification that throws (a wallet DB failure) requests a clean shutdown instead of escaping the
// notifier thread into std::terminate; a successful one doesn't; thread interruption still propagates.
TEST(FdDefense, WalletNotifyFailureShutsDownCleanly) {
noui_connect(); // the daemon connects these UI signals at startup; the error path shows a message box
fRequestShutdown = false;
EXPECT_TRUE(NotifyWalletsOrShutdown([] {}));
EXPECT_FALSE(ShutdownRequested());
EXPECT_FALSE(NotifyWalletsOrShutdown([] { throw std::runtime_error("CWalletDB::ListAccountCreditDebit(): error scanning DB"); }));
EXPECT_TRUE(ShutdownRequested());
fRequestShutdown = false;
EXPECT_FALSE(NotifyWalletsOrShutdown([] { throw 42; }));
EXPECT_TRUE(ShutdownRequested());
fRequestShutdown = false;
EXPECT_THROW(NotifyWalletsOrShutdown([] { throw boost::thread_interrupted(); }), boost::thread_interrupted);
EXPECT_FALSE(ShutdownRequested());
}
src/gtest/test_net_closesocket.cpp
+50
View File
@@ -0,0 +1,50 @@
// Copyright (c) 2016-2024 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
#include <gtest/gtest.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <unistd.h>
#include "net.h"
#include "util/sock.h"
// Regression test for the wolfSSL CloseSocket macro (see net.h): closing a CNode's socket must invalidate
// hSocket, so neither a repeated CloseSocketDisconnect nor ~CNode can close that fd number again once another
// owner (the wallet's BDB files, a block file, an RPC socket) has been handed it.
TEST(NetCloseSocket, ClosedNodeNeverClosesAReusedFdNumber) {
int pair[2];
ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_STREAM, 0, pair));
const int number = pair[0];
CNode* node = new CNode(number, CAddress(), "", true); // inbound: sends no version message
node->CloseSocketDisconnect();
EXPECT_EQ(INVALID_SOCKET, node->hSocket) << "CloseSocketDisconnect left the closed fd number in hSocket";
EXPECT_EQ(-1, fcntl(number, F_GETFD)) << "the node's socket wasn't closed";
// Another owner gets the freed number, as the kernel's lowest-free-fd rule does in a busy node.
const int other = open("/dev/null", O_RDONLY);
ASSERT_GE(other, 0);
ASSERT_EQ(number, dup2(other, number));
node->CloseSocketDisconnect(); // e.g. a send error after the disconnect was already handled
delete node; // ~CNode
EXPECT_NE(-1, fcntl(number, F_GETFD)) << "a closed CNode closed the fd number's new owner";
close(number);
close(other);
close(pair[1]);
}
// util/sock.cpp's CloseSocket is the one that must be called: it closes once and invalidates.
TEST(NetCloseSocket, CloseSocketInvalidates) {
int pair[2];
ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_STREAM, 0, pair));
SOCKET s = pair[0];
EXPECT_TRUE(CloseSocket(s));
EXPECT_EQ(INVALID_SOCKET, s);
EXPECT_FALSE(CloseSocket(s)) << "a second close of an invalidated handle must be a no-op";
close(pair[1]);
}
src/gtest/test_operation_status.cpp
-936
View File
@@ -1,936 +0,0 @@
// Copyright (c) 2016-2026 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
//
// z_getoperationstatus / z_getoperationresult / z_listoperationids after they stopped sorting
// whole UniValues, and after z_getoperationstatus and z_listoperationids stopped taking
// cs_main + cs_wallet. These tests show that:
// * both status calls return byte for byte what the original algorithm (copied below from
// c1040028e) returns, for queued, executing, successful, failed and cancelled operations,
// with and without id filters, with unknown and duplicate ids, with more than 16
// operations and with creation_time ties. std::sort reorders equal keys once there are
// more than 16 of them, so the order of same-second operations is part of the contract;
// * z_getoperationresult removes exactly the finished operations that it reports;
// * z_getoperationstatus and z_listoperationids answer while another thread holds
// cs_main + cs_wallet, and z_getoperationresult still waits for those locks;
// * AsyncRPCQueue::popOperationForId hands an operation to exactly one of several concurrent
// callers, so concurrent removers never report the same operation twice.
//
// No worker runs on the shared async queue in hush-gtest, so every operation keeps the state the
// test gives it. Without pwalletMain the handlers throw "Method not found", and
// z_getoperationresult locks pwalletMain->cs_wallet, so each test installs an empty in-memory
// CWallet. These tests use TEST() and a TestEnv guard rather than a fixture, because a fixture
// class cannot be named OperationStatus: that name is the operation state enum in
// asyncrpcoperation.h.
#include <gtest/gtest.h>
#include <univalue.h>
#include <algorithm>
#include <atomic>
#include <chrono>
#include <condition_variable>
#include <ctime>
#include <future>
#include <map>
#include <memory>
#include <mutex>
#include <set>
#include <stdexcept>
#include <string>
#include <thread>
#include <vector>
#include "asyncrpcoperation.h"
#include "asyncrpcqueue.h"
#include "init.h"
#include "main.h"
#include "pubkey.h"
#include "rpc/server.h"
#include "sync.h"
#include "wallet/wallet.h"
// Defined in wallet/rpcwallet.cpp (non-static, no header). Called directly so that removers can
// run without the cs_main + cs_wallet that z_getoperationresult takes.
extern UniValue z_getoperationstatus_IMPL(const UniValue& params, bool fRemoveFinishedOperations);
namespace {
// Bound for every wait that must succeed, so a regression fails instead of hanging.
const std::chrono::milliseconds kGuardTimeout(5000);
const char* const kQueueNotEmpty = "the shared async queue already holds operations from elsewhere";
const char* const kPoolFailed = "could not create operations with controlled creation_time ties";
enum FakeKind {
FAKE_QUEUED = 0,
FAKE_EXECUTING,
FAKE_SUCCESS,
FAKE_FAILED,
FAKE_CANCELLED,
FAKE_KINDS
};
// An operation whose state is set by the test and never changes after it is queued: main() does
// nothing, and nothing runs it anyway. getStatus() appends method/params exactly the way
// z_sendmany, z_mergetoaddress and z_shieldcoinbase append their contextinfo_.
class FixedStateOperation : public AsyncRPCOperation {
public:
virtual void main() {}
virtual UniValue getStatus() const {
UniValue v = AsyncRPCOperation::getStatus();
if (contextinfo_.isNull()) {
return v;
}
UniValue obj = v.get_obj();
obj.push_back(Pair("method", "z_fixedstate"));
obj.push_back(Pair("params", contextinfo_));
return obj;
}
// Only called before the operation is queued, like a real operation's constructor.
void SetContext(const UniValue& context) {
contextinfo_ = context;
}
void ForceExecuting() {
set_state(OperationStatus::EXECUTING);
}
// Same order as the real operations: stop the clock, then publish the result and SUCCESS.
void ForceSuccess(const UniValue& result) {
start_execution_clock();
stop_execution_clock();
set_result(result);
set_state(OperationStatus::SUCCESS);
}
void ForceFailed(int code, const std::string& message) {
set_error_code(code);
set_error_message(message);
set_state(OperationStatus::FAILED);
}
private:
UniValue contextinfo_;
};
std::shared_ptr<FixedStateOperation> MakeOperation(int kind, int tag)
{
std::shared_ptr<FixedStateOperation> op(new FixedStateOperation());
if (tag % 3 == 0) {
UniValue context(UniValue::VOBJ);
context.push_back(Pair("fromaddress", "zs1fixedstate"));
context.push_back(Pair("minconf", tag));
op->SetContext(context);
}
switch (kind) {
case FAKE_QUEUED:
break;
case FAKE_EXECUTING:
op->ForceExecuting();
break;
case FAKE_SUCCESS:
if (tag % 2) {
UniValue result(UniValue::VOBJ);
result.push_back(Pair("txid", "txid-" + std::to_string(tag)));
op->ForceSuccess(result);
} else {
op->ForceSuccess(UniValue("done"));
}
break;
case FAKE_FAILED:
op->ForceFailed(-6, "boom " + std::to_string(tag));
break;
case FAKE_CANCELLED:
op->cancel(); // READY -> CANCELLED, the same path shutdown and wallet.cpp use
break;
}
return op;
}
bool IsFinished(const AsyncRPCOperation& op)
{
return op.isSuccess() || op.isFailed() || op.isCancelled();
}
// The z_getoperationstatus_IMPL of c1040028e, verbatim except that the LOCK2 and the
// popOperationForId call are left out. Evaluated on an unchanged queue just before the call under
// test, it sees the same getAllOperationIds() enumeration and returns what the original returned:
// the original removing pass popped each operation only after reading it, so it reported exactly
// what this non-removing pass reports.
UniValue ReferenceOperationStatus(const UniValue& params, bool fRemoveFinishedOperations)
{
std::set<AsyncRPCOperationId> filter;
if (params.size()==1) {
UniValue ids = params[0].get_array();
for (const UniValue & v : ids.getValues()) {
filter.insert(v.get_str());
}
}
bool useFilter = (filter.size()>0);
UniValue ret(UniValue::VARR);
std::shared_ptr<AsyncRPCQueue> q = getAsyncRPCQueue();
std::vector<AsyncRPCOperationId> ids = q->getAllOperationIds();
for (auto id : ids) {
if (useFilter && !filter.count(id))
continue;
std::shared_ptr<AsyncRPCOperation> operation = q->getOperationForId(id);
if (!operation) {
continue;
}
UniValue obj = operation->getStatus();
std::string s = obj["status"].get_str();
if (fRemoveFinishedOperations) {
// Caller is only interested in retrieving finished results
if ("success"==s || "failed"==s || "cancelled"==s) {
ret.push_back(obj);
}
} else {
ret.push_back(obj);
}
}
std::vector<UniValue> arrTmp = ret.getValues();
// sort results chronologically by creation_time
std::sort(arrTmp.begin(), arrTmp.end(), [](UniValue a, UniValue b) -> bool {
const int64_t t1 = find_value(a.get_obj(), "creation_time").get_int64();
const int64_t t2 = find_value(b.get_obj(), "creation_time").get_int64();
return t1 < t2;
});
ret.clear();
ret.setArray();
ret.push_backV(arrTmp);
return ret;
}
// The body of c1040028e's z_listoperationids after its argument check, verbatim without the LOCK2.
UniValue ReferenceListOperationIds(const UniValue& params)
{
std::string filter;
bool useFilter = false;
if (params.size()==1) {
filter = params[0].get_str();
useFilter = true;
}
UniValue ret(UniValue::VARR);
std::shared_ptr<AsyncRPCQueue> q = getAsyncRPCQueue();
std::vector<AsyncRPCOperationId> ids = q->getAllOperationIds();
for (auto id : ids) {
std::shared_ptr<AsyncRPCOperation> operation = q->getOperationForId(id);
if (!operation) {
continue;
}
std::string state = operation->getStateAsString();
if (useFilter && filter.compare(state)!=0)
continue;
ret.push_back(id);
}
return ret;
}
// Results and errors as one comparable string. These helpers make no gtest calls, so helper
// threads can use them.
std::string CallOutcome(rpcfn_type fn, const UniValue& params)
{
try {
return fn(params, false, CPubKey()).write();
} catch (const UniValue& objError) {
return "rpc error: " + objError.write();
} catch (const std::exception& e) {
return std::string("exception: ") + e.what();
}
}
std::string ReferenceOutcome(const UniValue& params, bool fRemoveFinishedOperations)
{
try {
return ReferenceOperationStatus(params, fRemoveFinishedOperations).write();
} catch (const UniValue& objError) {
return "rpc error: " + objError.write();
} catch (const std::exception& e) {
return std::string("exception: ") + e.what();
}
}
std::string ReferenceListOutcome(const UniValue& params)
{
try {
return ReferenceListOperationIds(params).write();
} catch (const UniValue& objError) {
return "rpc error: " + objError.write();
} catch (const std::exception& e) {
return std::string("exception: ") + e.what();
}
}
// Byte comparison that reports where two (possibly very long) JSON texts first differ.
::testing::AssertionResult SameText(const std::string& original, const std::string& patched)
{
if (original == patched)
return ::testing::AssertionSuccess();
size_t i = 0;
while (i < original.size() && i < patched.size() && original[i] == patched[i])
i++;
const size_t from = i < 40 ? 0 : i - 40;
return ::testing::AssertionFailure()
<< "outputs differ at byte " << i << " (lengths " << original.size() << " / " << patched.size() << ")"
<< "\n original: ..." << original.substr(from, 120)
<< "\n patched: ..." << patched.substr(from, 120);
}
UniValue NoParams()
{
return UniValue(UniValue::VARR);
}
// [[id, ...]]
UniValue IdParams(const std::vector<std::string>& ids)
{
UniValue arr(UniValue::VARR);
for (size_t i = 0; i < ids.size(); i++)
arr.push_back(ids[i]);
UniValue params(UniValue::VARR);
params.push_back(arr);
return params;
}
// The ids a params array asks for. Empty means "all", as in the handlers.
std::set<std::string> RequestedIds(const UniValue& params)
{
std::set<std::string> ids;
if (params.size() == 1) {
const std::vector<UniValue>& v = params[0].get_array().getValues();
for (size_t i = 0; i < v.size(); i++)
ids.insert(v[i].get_str());
}
return ids;
}
std::set<std::string> ListedIds()
{
UniValue listed = z_listoperationids(NoParams(), false, CPubKey());
std::set<std::string> ids;
for (size_t i = 0; i < listed.size(); i++)
ids.insert(listed[i].get_str());
return ids;
}
// Installs an empty in-memory wallet as pwalletMain for one test, and when it goes out of scope
// removes every operation the test added to the shared queue and restores pwalletMain.
class TestEnv {
public:
TestEnv() : queue_(getAsyncRPCQueue()), savedWallet_(pwalletMain) {
wasEmpty_ = queue_->getAllOperationIds().empty();
pwalletMain = &wallet_;
}
~TestEnv() {
RemoveAll();
pwalletMain = savedWallet_;
}
bool QueueWasEmpty() const { return wasEmpty_; }
const std::shared_ptr<AsyncRPCQueue>& Queue() const { return queue_; }
// False if the queue refused an operation.
bool AddAll(const std::vector<std::shared_ptr<FixedStateOperation> >& ops) {
for (size_t i = 0; i < ops.size(); i++) {
if (!queue_->addOperation(ops[i]))
return false;
added_.push_back(ops[i]->getId());
}
return true;
}
void RemoveAll() {
for (size_t i = 0; i < added_.size(); i++)
queue_->popOperationForId(added_[i]);
added_.clear();
}
private:
std::shared_ptr<AsyncRPCQueue> queue_;
CWallet wallet_;
CWallet* savedWallet_;
bool wasEmpty_;
std::vector<AsyncRPCOperationId> added_;
};
// Holds cs_main and pwalletMain->cs_wallet on a helper thread, as a long wallet scan does, until
// Release() or destruction.
class WalletLockHolder {
public:
WalletLockHolder() : held_(false), release_(false), thread_(&WalletLockHolder::Run, this) {}
~WalletLockHolder() {
Release();
thread_.join();
}
bool WaitUntilHeld(std::chrono::milliseconds timeout) {
std::unique_lock<std::mutex> lk(m_);
return cv_.wait_for(lk, timeout, [this]() { return held_; });
}
void Release() {
{
std::lock_guard<std::mutex> guard(m_);
release_ = true;
}
cv_.notify_all();
}
private:
void Run() {
LOCK2(cs_main, pwalletMain->cs_wallet);
std::unique_lock<std::mutex> lk(m_);
held_ = true;
cv_.notify_all();
cv_.wait(lk, [this]() { return release_; });
}
std::mutex m_;
std::condition_variable cv_;
bool held_;
bool release_;
std::thread thread_; // last, so that everything Run() touches exists before it starts
};
// creation_time is time(NULL) at construction and cannot be set, so ties and distinct times are
// made by creating operations inside chosen seconds. Waits in 10 ms steps for the next second.
void WaitForNextSecond()
{
const time_t start = time(NULL);
for (int i = 0; i < 150 && time(NULL) == start; i++)
std::this_thread::sleep_for(std::chrono::milliseconds(10));
}
bool AllCreatedInOneSecond(const std::vector<std::shared_ptr<FixedStateOperation> >& ops)
{
for (size_t i = 1; i < ops.size(); i++) {
if (ops[i]->getCreationTime() != ops[0]->getCreationTime())
return false;
}
return true;
}
const int kPoolGroups = 3;
const size_t kPoolGroupSize = 300;
// kPoolGroups groups of operations in all five states. Every operation in a group has the same
// creation_time, and the groups' creation_times strictly increase.
struct OperationPool {
bool ok;
std::vector<std::vector<std::shared_ptr<FixedStateOperation> > > groups;
};
OperationPool BuildPool()
{
OperationPool pool;
pool.ok = true;
for (int g = 0; g < kPoolGroups && pool.ok; g++) {
std::vector<std::shared_ptr<FixedStateOperation> > group;
bool tied = false;
for (int attempt = 0; attempt < 3 && !tied; attempt++) {
if (g > 0 || attempt > 0)
WaitForNextSecond();
group.clear();
for (size_t i = 0; i < kPoolGroupSize; i++) {
group.push_back(MakeOperation(static_cast<int>((i + 2 * g) % FAKE_KINDS),
static_cast<int>(g * kPoolGroupSize + i)));
}
tied = AllCreatedInOneSecond(group);
}
const bool later = pool.groups.empty() ||
group.front()->getCreationTime() > pool.groups.back().front()->getCreationTime();
pool.ok = tied && later;
pool.groups.push_back(group);
}
return pool;
}
// Built once (it waits for up to about two second boundaries) and shared by the tests that need
// it. The operations are never modified after they are built.
const OperationPool& SharedPool()
{
static const OperationPool pool = BuildPool();
return pool;
}
struct Population {
std::string name;
std::vector<std::shared_ptr<FixedStateOperation> > ops;
};
// For each size: "ties-K", K operations from one second, and "mixed-K", K operations spread
// round robin over three seconds (ties within each second). 16 is the largest size that
// libstdc++'s std::sort handles with insertion sort alone; from 17 on, introsort reorders equal keys.
std::vector<Population> Populations(const OperationPool& pool)
{
static const size_t sizes[] = {0, 1, 2, 16, 17, 40, 300};
std::vector<Population> pops;
for (size_t s = 0; s < sizeof(sizes) / sizeof(sizes[0]); s++) {
const size_t k = sizes[s];
Population ties;
ties.name = "ties-" + std::to_string(k);
ties.ops.assign(pool.groups[1].begin(), pool.groups[1].begin() + k);
pops.push_back(ties);
Population mixed;
mixed.name = "mixed-" + std::to_string(k);
for (size_t i = 0; i < k; i++)
mixed.ops.push_back(pool.groups[i % kPoolGroups][i / kPoolGroups]);
pops.push_back(mixed);
}
return pops;
}
// Arguments for z_getoperationstatus/z_getoperationresult over a population: no argument, an
// empty list (no filter), an unknown id, two ids, a duplicated id, every other id plus an
// unknown one, and every id.
std::vector<UniValue> StatusParamSets(const std::vector<std::shared_ptr<FixedStateOperation> >& ops)
{
std::vector<UniValue> sets;
sets.push_back(NoParams());
sets.push_back(IdParams(std::vector<std::string>()));
sets.push_back(IdParams(std::vector<std::string>(1, "opid-unknown")));
if (!ops.empty()) {
std::vector<std::string> two;
two.push_back(ops.front()->getId());
two.push_back(ops.back()->getId());
sets.push_back(IdParams(two));
sets.push_back(IdParams(std::vector<std::string>(2, ops.front()->getId())));
std::vector<std::string> half, all;
for (size_t i = 0; i < ops.size(); i++) {
all.push_back(ops[i]->getId());
if (i % 2 == 0)
half.push_back(ops[i]->getId());
}
half.push_back("opid-unknown");
sets.push_back(IdParams(half));
sets.push_back(IdParams(all));
}
return sets;
}
// Pops every id once, as fast as it can, after the start signal.
void PopEach(AsyncRPCQueue* q, const std::vector<AsyncRPCOperationId>* ids,
const std::atomic<bool>* go, std::vector<AsyncRPCOperationId>* out)
{
while (!go->load())
std::this_thread::yield();
for (size_t i = 0; i < ids->size(); i++) {
if (q->popOperationForId((*ids)[i]) != nullptr)
out->push_back((*ids)[i]);
}
}
// Removes finished operations until the queue is empty (or kGuardTimeout passes) and records
// every id it was handed. If direct is true it calls the IMPL without cs_main/cs_wallet, so only
// the queue's atomic pop and the IMPL's checked pop keep it apart from the other removers.
// Otherwise it calls the z_getoperationresult handler, which takes the locks.
void DrainFinished(bool direct, const std::atomic<bool>* go, std::vector<std::string>* out)
{
while (!go->load())
std::this_thread::yield();
const UniValue params(UniValue::VARR);
std::shared_ptr<AsyncRPCQueue> q = getAsyncRPCQueue();
const std::chrono::steady_clock::time_point deadline = std::chrono::steady_clock::now() + kGuardTimeout;
while (std::chrono::steady_clock::now() < deadline) {
UniValue ret;
try {
ret = direct ? z_getoperationstatus_IMPL(params, true)
: z_getoperationresult(params, false, CPubKey());
for (size_t i = 0; i < ret.size(); i++)
out->push_back(find_value(ret[i], "id").get_str());
} catch (...) {
out->push_back("<exception>");
return;
}
if (ret.empty() && q->getAllOperationIds().empty())
return;
}
}
// Polls z_getoperationstatus while the removers run. Counts results that break its contract
// (only the test's finished operations exist): an unfinished status, a repeated id, or
// creation_time out of order.
void PollStatus(const std::atomic<bool>* go, const std::atomic<bool>* stop, int* faults)
{
while (!go->load())
std::this_thread::yield();
const UniValue params(UniValue::VARR);
while (!stop->load()) {
try {
UniValue ret = z_getoperationstatus(params, false, CPubKey());
std::set<std::string> ids;
int64_t last = 0;
for (size_t i = 0; i < ret.size(); i++) {
const std::string id = find_value(ret[i], "id").get_str();
const std::string status = find_value(ret[i], "status").get_str();
const int64_t t = find_value(ret[i], "creation_time").get_int64();
if (!ids.insert(id).second || t < last ||
(status != "success" && status != "failed" && status != "cancelled"))
(*faults)++;
last = t;
}
} catch (...) {
(*faults)++;
}
}
}
} // namespace
// The read path returns exactly what the original algorithm returned, including the order of
// same-second operations, and removes nothing.
TEST(OperationStatus, StatusMatchesOriginalAlgorithm)
{
const OperationPool& pool = SharedPool();
ASSERT_TRUE(pool.ok) << kPoolFailed;
TestEnv env;
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
const std::vector<Population> pops = Populations(pool);
for (size_t p = 0; p < pops.size(); p++) {
ASSERT_TRUE(env.AddAll(pops[p].ops));
const std::vector<UniValue> paramSets = StatusParamSets(pops[p].ops);
for (size_t i = 0; i < paramSets.size(); i++) {
EXPECT_TRUE(SameText(ReferenceOutcome(paramSets[i], false),
CallOutcome(&z_getoperationstatus, paramSets[i])))
<< pops[p].name << " params=" << paramSets[i].write();
}
EXPECT_EQ(pops[p].ops.size(), env.Queue()->getAllOperationIds().size()) << pops[p].name;
env.RemoveAll();
}
#if defined(__GLIBCXX__)
// Check that the comparison above can fail. With 40 operations in one second, libstdc++'s
// introsort reorders the equal keys, so the original output is not the enumeration order.
// A (creation_time, index) comparator would keep the enumeration order and fail above.
std::vector<std::shared_ptr<FixedStateOperation> > tied(pool.groups[1].begin(), pool.groups[1].begin() + 40);
ASSERT_TRUE(env.AddAll(tied));
const UniValue ref = ReferenceOperationStatus(NoParams(), false);
std::vector<std::string> reported;
for (size_t i = 0; i < ref.size(); i++)
reported.push_back(find_value(ref[i], "id").get_str());
const std::vector<std::string> enumerated = env.Queue()->getAllOperationIds();
EXPECT_EQ(enumerated.size(), reported.size());
EXPECT_NE(enumerated, reported) << "40 tied operations kept their enumeration order";
env.RemoveAll();
#endif
}
// The removing path reports exactly what the original reported, removes exactly the finished
// operations it was asked about, and leaves the rest visible to the other two calls.
TEST(OperationStatus, ResultMatchesOriginalAlgorithm)
{
const OperationPool& pool = SharedPool();
ASSERT_TRUE(pool.ok) << kPoolFailed;
TestEnv env;
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
const std::vector<Population> pops = Populations(pool);
for (size_t p = 0; p < pops.size(); p++) {
const std::vector<UniValue> paramSets = StatusParamSets(pops[p].ops);
for (size_t i = 0; i < paramSets.size(); i++) {
const UniValue& params = paramSets[i];
const std::string where = pops[p].name + " params=" + params.write();
ASSERT_TRUE(env.AddAll(pops[p].ops));
const std::string ref = ReferenceOutcome(params, true);
EXPECT_TRUE(SameText(ref, CallOutcome(&z_getoperationresult, params))) << where;
const std::set<std::string> requested = RequestedIds(params);
std::set<std::string> remaining;
int wrong = 0;
for (size_t j = 0; j < pops[p].ops.size(); j++) {
const std::shared_ptr<FixedStateOperation>& op = pops[p].ops[j];
const bool selected = requested.empty() || requested.count(op->getId()) > 0;
const bool present = env.Queue()->getOperationForId(op->getId()) != nullptr;
if (present != !(IsFinished(*op) && selected))
wrong++;
if (present)
remaining.insert(op->getId());
}
EXPECT_EQ(0, wrong) << where << ": operations removed or kept wrongly";
// Nothing is handed out twice, and the survivors are still listed and reported.
EXPECT_EQ("[]", CallOutcome(&z_getoperationresult, params)) << where;
EXPECT_EQ(remaining, ListedIds()) << where;
EXPECT_TRUE(SameText(ReferenceOutcome(NoParams(), false),
CallOutcome(&z_getoperationstatus, NoParams()))) << where;
env.RemoveAll();
}
}
}
TEST(OperationStatus, ListOperationIdsMatchesOriginalAlgorithm)
{
const OperationPool& pool = SharedPool();
ASSERT_TRUE(pool.ok) << kPoolFailed;
TestEnv env;
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
const char* const filters[] = {NULL, "queued", "executing", "success", "failed", "cancelled", "bogus"};
const std::vector<Population> pops = Populations(pool);
for (size_t p = 0; p < pops.size(); p++) {
ASSERT_TRUE(env.AddAll(pops[p].ops));
for (size_t f = 0; f < sizeof(filters) / sizeof(filters[0]); f++) {
UniValue params(UniValue::VARR);
if (filters[f] != NULL)
params.push_back(filters[f]);
EXPECT_TRUE(SameText(ReferenceListOutcome(params), CallOutcome(&z_listoperationids, params)))
<< pops[p].name << " params=" << params.write();
}
env.RemoveAll();
}
}
// Malformed arguments fail exactly as before and remove nothing; no wallet still means
// "Method not found".
TEST(OperationStatus, ParameterErrorsUnchanged)
{
TestEnv env;
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
std::vector<std::shared_ptr<FixedStateOperation> > ops;
for (int k = 0; k < FAKE_KINDS; k++)
ops.push_back(MakeOperation(k, k));
ASSERT_TRUE(env.AddAll(ops));
UniValue notArray(UniValue::VARR);
notArray.push_back("not_an_array");
UniValue inner(UniValue::VARR);
inner.push_back(ops[FAKE_SUCCESS]->getId());
inner.push_back(1);
UniValue notString(UniValue::VARR);
notString.push_back(inner);
EXPECT_EQ("exception: JSON value is not an array as expected", ReferenceOutcome(notArray, false));
EXPECT_EQ("exception: JSON value is not a string as expected", ReferenceOutcome(notString, false));
EXPECT_EQ(ReferenceOutcome(notArray, false), CallOutcome(&z_getoperationstatus, notArray));
EXPECT_EQ(ReferenceOutcome(notString, false), CallOutcome(&z_getoperationstatus, notString));
EXPECT_EQ(ReferenceOutcome(notArray, true), CallOutcome(&z_getoperationresult, notArray));
EXPECT_EQ(ReferenceOutcome(notString, true), CallOutcome(&z_getoperationresult, notString));
UniValue tooMany(UniValue::VARR);
tooMany.push_back(UniValue(UniValue::VARR));
tooMany.push_back("toomanyargs");
EXPECT_THROW(z_getoperationstatus(tooMany, false, CPubKey()), std::runtime_error);
EXPECT_THROW(z_getoperationresult(tooMany, false, CPubKey()), std::runtime_error);
EXPECT_THROW(z_listoperationids(tooMany, false, CPubKey()), std::runtime_error);
EXPECT_EQ(ops.size(), env.Queue()->getAllOperationIds().size());
CWallet* wallet = pwalletMain;
pwalletMain = NULL;
EXPECT_THROW(z_getoperationstatus(NoParams(), false, CPubKey()), UniValue);
EXPECT_THROW(z_getoperationresult(NoParams(), false, CPubKey()), UniValue);
EXPECT_THROW(z_listoperationids(NoParams(), false, CPubKey()), UniValue);
pwalletMain = wallet;
}
// While another thread holds cs_main + cs_wallet, z_getoperationstatus (unfiltered and filtered,
// as the GUI polls it) and z_listoperationids still answer, with unchanged output, and
// z_getoperationresult waits for the locks and then answers as before.
TEST(OperationStatus, StatusDoesNotWaitForWalletLocks)
{
TestEnv env;
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
std::vector<std::shared_ptr<FixedStateOperation> > ops;
for (int k = 0; k < FAKE_KINDS; k++)
ops.push_back(MakeOperation(k, k));
ASSERT_TRUE(env.AddAll(ops));
const UniValue filtered = IdParams(std::vector<std::string>(1, ops[FAKE_SUCCESS]->getId()));
const std::string refStatus = ReferenceOutcome(NoParams(), false);
const std::string refFiltered = ReferenceOutcome(filtered, false);
const std::string refList = ReferenceListOutcome(NoParams());
const std::string refResult = ReferenceOutcome(NoParams(), true);
// Declared before the holder: on an early return the holder is destroyed first, which
// releases the locks, and only then do the futures' destructors wait for their calls.
std::future<std::string> status, statusFiltered, listed, result;
WalletLockHolder holder;
ASSERT_TRUE(holder.WaitUntilHeld(kGuardTimeout)) << "helper thread could not take cs_main + cs_wallet";
status = std::async(std::launch::async, CallOutcome, &z_getoperationstatus, NoParams());
statusFiltered = std::async(std::launch::async, CallOutcome, &z_getoperationstatus, filtered);
listed = std::async(std::launch::async, CallOutcome, &z_listoperationids, NoParams());
result = std::async(std::launch::async, CallOutcome, &z_getoperationresult, NoParams());
const bool statusDone = status.wait_for(kGuardTimeout) == std::future_status::ready;
const bool filteredDone = statusFiltered.wait_for(kGuardTimeout) == std::future_status::ready;
const bool listDone = listed.wait_for(kGuardTimeout) == std::future_status::ready;
// The locks are still held, so a z_getoperationresult that takes them cannot have finished.
const bool resultWaited = result.wait_for(std::chrono::milliseconds(100)) == std::future_status::timeout;
holder.Release();
const bool resultDone = result.wait_for(kGuardTimeout) == std::future_status::ready;
EXPECT_TRUE(statusDone) << "z_getoperationstatus waited for cs_main/cs_wallet";
EXPECT_TRUE(filteredDone) << "filtered z_getoperationstatus waited for cs_main/cs_wallet";
EXPECT_TRUE(listDone) << "z_listoperationids waited for cs_main/cs_wallet";
EXPECT_TRUE(resultWaited) << "z_getoperationresult no longer waits for cs_main/cs_wallet";
EXPECT_TRUE(resultDone) << "z_getoperationresult did not finish after the locks were released";
if (statusDone) {
EXPECT_EQ(refStatus, status.get());
}
if (filteredDone) {
EXPECT_EQ(refFiltered, statusFiltered.get());
}
if (listDone) {
EXPECT_EQ(refList, listed.get());
}
if (resultDone) {
EXPECT_EQ(refResult, result.get());
}
// The result call removed the three finished operations and kept the queued and executing ones.
std::set<std::string> expected;
expected.insert(ops[FAKE_QUEUED]->getId());
expected.insert(ops[FAKE_EXECUTING]->getId());
EXPECT_EQ(expected, ListedIds());
}
// SC-1: popOperationForId looks up and erases under one hold of the queue lock, so when several
// threads pop the same ids each operation goes to exactly one of them. The old two-step pop
// could hand the same operation to two callers; this catches that probabilistically.
TEST(OperationStatus, PopOperationForIdHandsEachOperationToOneCaller)
{
{
std::shared_ptr<AsyncRPCQueue> q = std::make_shared<AsyncRPCQueue>();
std::shared_ptr<FixedStateOperation> op = MakeOperation(FAKE_SUCCESS, 1);
ASSERT_TRUE(q->addOperation(op));
EXPECT_TRUE(q->popOperationForId("opid-unknown") == nullptr);
EXPECT_TRUE(q->popOperationForId(op->getId()).get() == op.get());
EXPECT_TRUE(q->getOperationForId(op->getId()) == nullptr);
EXPECT_TRUE(q->popOperationForId(op->getId()) == nullptr);
}
const int kThreads = 8;
const int kOps = 1000;
const int kRounds = 10;
for (int round = 0; round < kRounds; round++) {
// A private queue with no workers.
std::shared_ptr<AsyncRPCQueue> q = std::make_shared<AsyncRPCQueue>();
std::vector<AsyncRPCOperationId> ids;
for (int i = 0; i < kOps; i++) {
std::shared_ptr<AsyncRPCOperation> op(new FixedStateOperation());
ASSERT_TRUE(q->addOperation(op));
ids.push_back(op->getId());
}
std::atomic<bool> go(false);
std::vector<std::vector<AsyncRPCOperationId> > popped(kThreads);
std::vector<std::thread> threads;
for (int t = 0; t < kThreads; t++)
threads.push_back(std::thread(PopEach, q.get(), &ids, &go, &popped[t]));
go.store(true);
for (size_t t = 0; t < threads.size(); t++)
threads[t].join();
std::map<AsyncRPCOperationId, int> handed;
size_t total = 0;
for (int t = 0; t < kThreads; t++) {
for (size_t i = 0; i < popped[t].size(); i++) {
handed[popped[t][i]]++;
total++;
}
}
int wrong = 0;
for (size_t i = 0; i < ids.size(); i++) {
std::map<AsyncRPCOperationId, int>::const_iterator it = handed.find(ids[i]);
if (it == handed.end() || it->second != 1)
wrong++;
}
EXPECT_EQ(0, wrong) << "round " << round << ": operations not handed to exactly one caller";
EXPECT_EQ(static_cast<size_t>(kOps), total) << "round " << round;
EXPECT_TRUE(q->getAllOperationIds().empty()) << "round " << round;
}
}
// Removers through the locked handler and straight through the IMPL (no wallet locks) drain the
// same finished operations while two lock-free z_getoperationstatus pollers run. Every operation
// is reported exactly once, and no poll returns a malformed or unsorted list.
TEST(OperationStatus, ConcurrentRemoversReportEachOperationOnce)
{
TestEnv env;
ASSERT_TRUE(env.QueueWasEmpty()) << kQueueNotEmpty;
const int kOps = 200;
std::vector<std::shared_ptr<FixedStateOperation> > ops;
std::set<std::string> expected;
for (int i = 0; i < kOps; i++) {
ops.push_back(MakeOperation(FAKE_SUCCESS + i % 3, i)); // success, failed or cancelled
expected.insert(ops.back()->getId());
}
ASSERT_TRUE(env.AddAll(ops));
const int kRemovers = 8;
const int kReaders = 2;
std::atomic<bool> go(false);
std::atomic<bool> stop(false);
std::vector<std::vector<std::string> > removed(kRemovers);
std::vector<int> readerFaults(kReaders, 0);
std::vector<std::thread> removers, readers;
for (int t = 0; t < kRemovers; t++)
removers.push_back(std::thread(DrainFinished, t % 2 == 0, &go, &removed[t]));
for (int r = 0; r < kReaders; r++)
readers.push_back(std::thread(PollStatus, &go, &stop, &readerFaults[r]));
go.store(true);
for (size_t t = 0; t < removers.size(); t++)
removers[t].join();
stop.store(true);
for (size_t r = 0; r < readers.size(); r++)
readers[r].join();
std::map<std::string, int> reported;
size_t total = 0;
for (int t = 0; t < kRemovers; t++) {
for (size_t i = 0; i < removed[t].size(); i++) {
reported[removed[t][i]]++;
total++;
}
}
int duplicated = 0, unknown = 0;
for (std::map<std::string, int>::const_iterator it = reported.begin(); it != reported.end(); ++it) {
if (it->second != 1)
duplicated++;
if (!expected.count(it->first))
unknown++;
}
EXPECT_EQ(0, duplicated) << "an operation was reported by more than one remover";
EXPECT_EQ(0, unknown) << "a remover reported an unknown id or threw";
EXPECT_EQ(static_cast<size_t>(kOps), reported.size());
EXPECT_EQ(static_cast<size_t>(kOps), total);
EXPECT_TRUE(env.Queue()->getAllOperationIds().empty());
for (int r = 0; r < kReaders; r++)
EXPECT_EQ(0, readerFaults[r]) << "reader " << r;
}
// SC-4: the const status map with .at() names every state exactly as before.
TEST(OperationStatus, StateNamesUnchanged)
{
const char* const names[FAKE_KINDS] = {"queued", "executing", "success", "failed", "cancelled"};
for (int k = 0; k < FAKE_KINDS; k++) {
std::shared_ptr<FixedStateOperation> op = MakeOperation(k, k + 1);
EXPECT_EQ(std::string(names[k]), op->getStateAsString());
const UniValue status = op->getStatus();
EXPECT_EQ(std::string(names[k]), find_value(status, "status").get_str());
}
}
src/gtest/test_wallet_dbenv.cpp
+58
View File
@@ -0,0 +1,58 @@
// Copyright (c) 2016-2024 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
#include <gtest/gtest.h>
#include <boost/filesystem.hpp>
#include "wallet/db.h"
namespace {
// A fresh environment in a temp dir with one committed record in "wallet.dat", so database/ holds a log file.
class WalletDbEnv : public ::testing::Test {
protected:
boost::filesystem::path dir;
CDBEnv env;
void SetUp() override {
dir = boost::filesystem::temp_directory_path() / boost::filesystem::unique_path("dbenv-%%%%%%%%");
boost::filesystem::create_directories(dir);
ASSERT_TRUE(env.Open(dir));
Db db(env.dbenv, DB_CXX_NO_EXCEPTIONS);
ASSERT_EQ(0, db.open(NULL, "wallet.dat", "main", DB_BTREE, DB_CREATE | DB_AUTO_COMMIT, 0));
char k[] = "key", v[] = "value";
Dbt key(k, sizeof k), value(v, sizeof v);
ASSERT_EQ(0, db.put(NULL, &key, &value, 0));
ASSERT_EQ(0, db.close(0));
ASSERT_TRUE(boost::filesystem::exists(Log()));
}
void TearDown() override {
env.Close();
boost::system::error_code ec;
boost::filesystem::remove_all(dir, ec);
}
boost::filesystem::path Log() const { return dir / "database" / "log.0000000001"; }
};
} // namespace
TEST_F(WalletDbEnv, CleanShutdownRemovesTheLogs) {
env.Flush(true);
EXPECT_FALSE(boost::filesystem::exists(dir / "database"));
}
// BDB0060 PANIC (a page write failed, e.g. through an fd closed behind BDB's back): the logs may hold committed
// changes wallet.dat never got, and the next start's DB_RECOVER needs them.
TEST_F(WalletDbEnv, PanickedEnvironmentKeepsTheLogs) {
ASSERT_EQ(0, env.dbenv->set_flags(DB_PANIC_ENVIRONMENT, 1));
env.Flush(true);
EXPECT_TRUE(boost::filesystem::exists(Log()));
}
TEST_F(WalletDbEnv, FailedCheckpointKeepsTheLogs) {
env.CheckpointLSN("no-such-file.dat"); // lsn_reset fails
env.Flush(true);
EXPECT_TRUE(boost::filesystem::exists(Log()));
}
src/gtest/test_wallet_history.cpp
+816
View File
@@ -0,0 +1,816 @@
// Copyright (c) 2016-2024 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
#include <gtest/gtest.h>
#include <deque>
#include <utf8.h>
#include "chain.h"
#include "chainparams.h"
#include "coins.h"
#include "init.h"
#include "key_io.h"
#include "main.h"
#include "primitives/block.h"
#include "primitives/transaction.h"
#include "random.h"
#include "rpc/server.h"
#include "script/standard.h"
#include "utilmoneystr.h"
#include "wallet/wallet.h"
#include "wallet/rpchushwallet.h"
#include "zcash/Address.hpp"
#include "zcash/Note.hpp"
#include "zcash/NoteEncryption.hpp"
#include "zcash/zip32.h"
using namespace std;
using namespace libzcash;
// The history-RPC helpers (src/wallet/rpchushwallet.cpp) as of 69bb035e7, verbatim apart from the Legacy_
// prefix and static linkage: the reference the optimized helpers must reproduce byte for byte.
static void Legacy_zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) {
LOCK2(cs_main, pwalletMain->cs_wallet);
//Used to identify incomplete key sets
int vinCount = 0;
int vinSpendCount = 0;
int shieldedSpendCount = 0;
int shieldedSpendsSpentCount = 0;
//Check address
bool isTAddress = false;
bool isZsAddress = false;
CTxDestination tAddress = DecodeDestination(strAddress);
auto zAddress = DecodePaymentAddress(strAddress);
SaplingPaymentAddress zsAddress;
if (filterByAddress) {
if (IsValidDestination(tAddress))
isTAddress = true;
if (IsValidPaymentAddress(zAddress)) {
if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
isZsAddress = true;
}
}
}
// Transparent Inputs belonging to the wallet
UniValue tSpends(UniValue::VARR);
if (isTAddress || !filterByAddress) {
for (int i = 0; i < wtx.vin.size(); i++) {
vinCount++;
const CTxIn& txin = wtx.vin[i];
UniValue obj(UniValue::VOBJ);
CTxDestination address;
const CWalletTx* parent = pwalletMain->GetWalletTx(txin.prevout.hash);
if (parent != NULL) {
const CTxOut& parentOut = parent->vout[txin.prevout.n];
ExtractDestination(parentOut.scriptPubKey, address);
if(IsMine(*pwalletMain, address)){
vinSpendCount++;
totalSpends += CAmount(-parentOut.nValue);
obj.push_back(Pair("address",EncodeDestination(address)));
obj.push_back(Pair("scriptPubKey",HexStr(parentOut.scriptPubKey.begin(), parentOut.scriptPubKey.end())));
obj.push_back(Pair("amount",ValueFromAmount(-parentOut.nValue)));
obj.push_back(Pair("spendTxid",parent->GetHash().ToString()));
obj.push_back(Pair("spendVout",(int)txin.prevout.n));
CTxDestination filterAddress = DecodeDestination(strAddress);
if (address == tAddress || !filterByAddress) {
filteredSpends += CAmount(-parentOut.nValue);
tSpends.push_back(obj);
}
}
}
}
}
spends.push_back(Pair("transparentSpends",tSpends));
// Sapling Inputs belonging to the wallet
UniValue zsSpends(UniValue::VARR);
if (isZsAddress || !filterByAddress) {
for (int i = 0; i < wtx.vShieldedSpend.size(); i++) {
shieldedSpendCount++;
const SpendDescription& spendDesc = wtx.vShieldedSpend[i];
UniValue obj(UniValue::VOBJ);
SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier];
if (pwalletMain->IsSaplingNullifierFromMe(spendDesc.nullifier)) {
const CWalletTx* parent = pwalletMain->GetWalletTx(pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash);
const OutputDescription& output = parent->vShieldedOutput[op.n];
auto nd = pwalletMain->mapWallet[pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash].mapSaplingNoteData[op];
auto pt = libzcash::SaplingNotePlaintext::decrypt(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
if (pt) {
auto note = pt.get();
auto pa = nd.ivk.address(note.d);
auto address = pa.get();
shieldedSpendsSpentCount++;
totalSpends += CAmount(-note.value());
obj.push_back(Pair("address",EncodePaymentAddress(address)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(-note.value()))));
obj.push_back(Pair("spendTxid",parent->GetHash().ToString()));
obj.push_back(Pair("spendshieldedOutputIndex",(int)op.n));
if (address == zsAddress || !filterByAddress) {
filteredSpends += CAmount(-note.value());
zsSpends.push_back(obj);
}
}
}
}
}
spends.push_back(Pair("saplingSpends",zsSpends));
spends.push_back(Pair("totalSpends",ValueFromAmount(filteredSpends)));
if (!filterByAddress) {
if (filteredSpends != 0 && (vinCount != vinSpendCount || shieldedSpendCount != shieldedSpendsSpentCount)) {
spends.push_back(Pair("missingSpendingKeys", true));
spends.push_back(Pair("vinCount", vinCount));
spends.push_back(Pair("vinSpendCount", vinSpendCount));
spends.push_back(Pair("shieldedSpendCount", shieldedSpendCount));
spends.push_back(Pair("shieldedSpendsSpentCount", shieldedSpendsSpentCount));
} else {
spends.push_back(Pair("missingSpendingKeys", false));
}
}
}
static void Legacy_zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress) {
if(fZdebug)
fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
LOCK2(cs_main, pwalletMain->cs_wallet);
//Check address
bool isTAddress = false;
bool isZsAddress = false;
CTxDestination tAddress = DecodeDestination(strAddress);
auto zAddress = DecodePaymentAddress(strAddress);
SaplingPaymentAddress zsAddress;
if (filterByAddress) {
if (IsValidDestination(tAddress))
isTAddress = true;
if (IsValidPaymentAddress(zAddress)) {
if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
isZsAddress = true;
}
}
}
//Transparent Received txos belonging to the wallet
UniValue tReceived(UniValue::VARR);
if (isTAddress || !filterByAddress) {
for (int i = 0; i < wtx.vout.size(); i++) {
const CTxOut& txout = wtx.vout[i];
UniValue obj(UniValue::VOBJ);
CTxDestination address;
ExtractDestination(txout.scriptPubKey, address);
if(IsMine(*pwalletMain, address)){
obj.push_back(Pair("address",EncodeDestination(address)));
obj.push_back(Pair("scriptPubKey",HexStr(txout.scriptPubKey.begin(), txout.scriptPubKey.end())));
obj.push_back(Pair("amount",ValueFromAmount(txout.nValue)));
obj.push_back(Pair("vout", i));
if (address == tAddress || !filterByAddress) {
totalReceived += CAmount(txout.nValue);
tReceived.push_back(obj);
}
}
}
}
received.push_back(Pair("transparentReceived",tReceived));
//Sapling Sends belonging to the wallet
UniValue zsReceived(UniValue::VARR);
if (isZsAddress || !filterByAddress) {
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
UniValue obj(UniValue::VOBJ);
bool changeTx = false;
//Decrypt sapling incoming commitments using IVK
std::set<libzcash::SaplingPaymentAddress> addresses;
pwalletMain->GetSaplingPaymentAddresses(addresses);
for (auto addr : addresses) {
libzcash::SaplingExtendedSpendingKey extsk;
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext, extsk.expsk.full_viewing_key().in_viewing_key(), outputDesc.ephemeralKey, outputDesc.cm);
if (pt) {
auto note = pt.get();
auto pt_unwrapped = pt.get();
auto memo = pt_unwrapped.memo();
obj.push_back(Pair("address",EncodePaymentAddress(addr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value()))));
obj.push_back(Pair("memo", HexStr(memo)));
if (memo[0] <= 0xf4) {
auto end = std::find_if(memo.rbegin(), memo.rend(), [](unsigned char v) { return v != 0; });
std::string memoStr(memo.begin(), end.base());
if (utf8::is_valid(memoStr)) {
obj.push_back(Pair("memoStr", memoStr));
}
}
obj.push_back(Pair("shieldedOutputIndex",i));
//Check Change Status
if (wtx.vShieldedSpend.size()!=0) {
std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
nullifierSet = pwalletMain->GetNullifiersForAddresses({addr});
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
if (nullifierSet.count(std::make_pair(addr, spendDesc.nullifier))) {
changeTx = true;
}
}
}
obj.push_back(Pair("change",changeTx));
if (addr == zsAddress || !filterByAddress) {
totalReceived += CAmount(note.value());
zsReceived.push_back(obj);
}
}
}
}
}
}
received.push_back(Pair("saplingReceived",zsReceived));
received.push_back(Pair("totalReceived",ValueFromAmount(totalReceived)));
}
static void Legacy_zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress) {
LOCK2(cs_main, pwalletMain->cs_wallet);
//Used to identify incomplete key sets
int shieldedOutputCount = 0;
int shieldedOutputDecryptedCount = 0;
//Check address
bool isTAddress = false;
bool isZsAddress = false;
CTxDestination tAddress = DecodeDestination(strAddress);
auto zAddress = DecodePaymentAddress(strAddress);
SaplingPaymentAddress zsAddress;
if (filterByAddress) {
if (IsValidDestination(tAddress))
isTAddress = true;
if (IsValidPaymentAddress(zAddress)) {
if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
isZsAddress = true;
}
}
}
//All Transparent Sends in the transaction
UniValue tSends(UniValue::VARR);
if (isTAddress || !filterByAddress) {
for (int i = 0; i < wtx.vout.size(); i++) {
const CTxOut& txout = wtx.vout[i];
UniValue obj(UniValue::VOBJ);
CTxDestination address;
ExtractDestination(txout.scriptPubKey, address);
obj.push_back(Pair("address",EncodeDestination(address)));
obj.push_back(Pair("scriptPubKey",HexStr(txout.scriptPubKey.begin(), txout.scriptPubKey.end())));
obj.push_back(Pair("amount",ValueFromAmount(-txout.nValue)));
obj.push_back(Pair("vout", i));
if (address == tAddress || !filterByAddress) {
totalSends += CAmount(-txout.nValue);
tSends.push_back(obj);
}
}
}
sends.push_back(Pair("transparentSends",tSends));
//All Shielded Sends in the transaction
UniValue zsSends(UniValue::VARR);
if (isZsAddress || !filterByAddress) {
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
shieldedOutputCount++;
UniValue obj(UniValue::VOBJ);
bool changeTx = false;
//Decrypt sapling outgoing t to z transaction using HDseed
if (wtx.vShieldedSpend.size()==0) {
HDSeed seed;
if (pwalletMain->GetHDSeedForDerivation(seed)) {
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
outputDesc.outCiphertext,ovkForShieldingFromTaddr(seed),outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
if (opt) {
auto opt_unwrapped = opt.get();
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
if (pt) {
shieldedOutputDecryptedCount++;
auto pt_unwrapped = pt.get();
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value()))));
obj.push_back(Pair("shieldedOutputIndex",i));
obj.push_back(Pair("change",false));
if (sentAddr == zsAddress || !filterByAddress) {
totalSends += CAmount(-pt_unwrapped.value());
zsSends.push_back(obj);
}
}
}
}
//attempt Decryption of Outgoing Sapling using wallet extended spending keys
} else {
std::set<libzcash::SaplingPaymentAddress> addresses;
pwalletMain->GetSaplingPaymentAddresses(addresses);
for (auto addr : addresses) {
libzcash::SaplingExtendedSpendingKey extsk;
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
outputDesc.outCiphertext,extsk.expsk.full_viewing_key().ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
if (opt) {
auto opt_unwrapped = opt.get();
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
if (pt) {
auto pt_unwrapped = pt.get();
auto memo = pt_unwrapped.memo();
shieldedOutputDecryptedCount++;
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value()))));
obj.push_back(Pair("memo", HexStr(memo)));
if (memo[0] <= 0xf4) {
auto end = std::find_if(memo.rbegin(), memo.rend(), [](unsigned char v) { return v != 0; });
std::string memoStr(memo.begin(), end.base());
if (utf8::is_valid(memoStr)) {
obj.push_back(Pair("memoStr", memoStr));
}
}
obj.push_back(Pair("shieldedOutputIndex",i));
//Check Change Status
if (wtx.vShieldedSpend.size()!=0) {
std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
nullifierSet = pwalletMain->GetNullifiersForAddresses({sentAddr});
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
if (nullifierSet.count(std::make_pair(sentAddr, spendDesc.nullifier))) {
changeTx = true;
}
}
}
obj.push_back(Pair("change",changeTx));
if (sentAddr == zsAddress || !filterByAddress) {
totalSends += CAmount(-pt_unwrapped.value());
zsSends.push_back(obj);
}
}
}
}
}
}
}
}
sends.push_back(Pair("saplingSends",zsSends));
if (shieldedOutputCount != shieldedOutputDecryptedCount) {
sends.push_back(Pair("missingSaplingOVK", true));
} else {
sends.push_back(Pair("missingSaplingOVK", false));
}
sends.push_back(Pair("totalSends",ValueFromAmount(totalSends)));
}
static void Legacy_zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType) {
LOCK2(cs_main, pwalletMain->cs_wallet);
//Track total wallet spend and received
CAmount totalSpends = 0;
CAmount filteredSpends = 0;
CAmount totalReceived = 0;
CAmount totalSends = 0;
//Various Univalue to be added to the final transaction
UniValue spends(UniValue::VOBJ);
UniValue received(UniValue::VOBJ);
UniValue sends(UniValue::VOBJ);
UniValue tx(UniValue::VOBJ);
//Begin Compiling the Decrypted Transaction
tx.push_back(Pair("txid", wtx.GetHash().ToString()));
if(fZdebug)
fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
if (wtx.IsCoinBase())
{
tx.push_back(Pair("coinbase", true));
if (wtx.GetDepthInMainChain() < 1)
tx.push_back(Pair("category", "orphan"));
else if (wtx.GetBlocksToMaturity() > 0)
tx.push_back(Pair("category", "immature"));
else
tx.push_back(Pair("category", "generate"));
} else {
tx.push_back(Pair("coinbase", false));
tx.push_back(Pair("category", "standard"));
}
tx.push_back(Pair("blockhash", wtx.hashBlock.GetHex()));
tx.push_back(Pair("blockindex", wtx.nIndex));
int confirms = wtx.GetDepthInMainChain();
if(confirms > 0)
{
tx.push_back(Pair("blocktime", mapBlockIndex[wtx.hashBlock]->GetBlockTime()));
} else {
tx.push_back(Pair("blocktime", 0));
}
tx.push_back(Pair("expiryheight", (int64_t)wtx.nExpiryHeight));
tx.push_back(Pair("confirmations", confirms));
tx.push_back(Pair("time", wtx.GetTxTime()));
tx.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
//Wallet Conflicts
UniValue conflicts(UniValue::VARR);
BOOST_FOREACH(const uint256& conflict, wtx.GetConflicts())
conflicts.push_back(conflict.GetHex());
tx.push_back(Pair("walletconflicts", conflicts));
// Return Type used to determine what is included in the transaction
// 0 Spends, Received and spends
// 1 Sends
// 2 Received
// 3 Spends
// TODO - Setup Enum, maybe...
// Add Spends retrieved earlier
if (returnType != 2) {
Legacy_zsTxSpendsToJSON(wtx, spends, totalSpends, filteredSpends, strAddress, fBool);
if ((!fBool || filteredSpends != 0) && (returnType == 0 || returnType == 1)) {
tx.push_back(Pair("spends",spends));
}
}
// Get Received
if (returnType == 0 || returnType == 2) {
Legacy_zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool);
if (!fBool || totalReceived != 0) {
tx.push_back(Pair("received",received));
}
}
// Get Sends
if (returnType == 0 || returnType == 3) {
//Only include sends if there are spends that belong to the wallet.
if (totalSpends != 0 || fBool) {
Legacy_zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool);
}
if (!fBool || totalSends != 0) {
tx.push_back(Pair("sends",sends));
}
}
if ((returnType == 0 && (!fBool || filteredSpends != 0 || totalReceived != 0 || totalSends != 0))
|| (returnType == 1 && (!fBool || filteredSpends != 0))
|| (returnType == 2 && (!fBool || totalReceived != 0))
|| (returnType == 3 && (!fBool || totalSends != 0))) {
ret.push_back(tx);
}
}
namespace {
std::array<unsigned char, HUSH_MEMO_SIZE> Memo(const std::string& text)
{
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
memo.fill(0);
if (text == "empty") // the "no memo" encoding: 0xF6 then zeros
memo[0] = 0xF6;
else if (text == "binary") // starts below 0xF5 but isn't UTF-8
memo[0] = 0xC3, memo[1] = 0x28;
else
memcpy(memo.data(), text.data(), text.size());
return memo;
}
// A real encrypted Sapling output, and when ovk is given, an outCiphertext the sender can decrypt with it
// (the wallet never checks proofs or value commitments).
OutputDescription Output(const SaplingPaymentAddress& to, uint64_t value, const std::string& memo,
const boost::optional<uint256>& ovk = boost::none)
{
SaplingNote note(to, value);
auto enc = SaplingNotePlaintext(note, Memo(memo)).encrypt(note.pk_d);
assert(static_cast<bool>(enc));
OutputDescription od;
od.cv = GetRandHash();
od.cm = note.cm().get();
od.ephemeralKey = enc->second.get_epk();
od.encCiphertext = enc->first;
if (ovk) {
SaplingOutgoingPlaintext out(note.pk_d, enc->second.get_esk());
od.outCiphertext = out.encrypt(*ovk, od.cv, od.cm, enc->second);
} else {
GetRandBytes(od.outCiphertext.begin(), od.outCiphertext.size());
}
return od;
}
CTransaction SaplingTx(const std::vector<OutputDescription>& outputs, const std::vector<uint256>& spentNullifiers,
const std::vector<COutPoint>& vin = {})
{
CMutableTransaction mtx;
mtx.fOverwintered = true;
mtx.nVersion = SAPLING_TX_VERSION;
mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
mtx.vShieldedOutput = outputs;
for (const uint256& nf : spentNullifiers) {
SpendDescription sd;
sd.nullifier = nf;
mtx.vShieldedSpend.push_back(sd);
}
for (const COutPoint& prev : vin)
mtx.vin.push_back(CTxIn(prev));
return CTransaction(mtx);
}
SaplingExtendedSpendingKey Key(uint32_t account)
{
std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x6b);
HDSeed seed(raw);
return SaplingExtendedSpendingKey::Master(seed).Derive(account | ZIP32_HARDENED_KEY_LIMIT);
}
}
// A wallet whose txs exercise every branch of the history helpers: notes received on default and diversified
// addresses, by a viewing-key-only address and by a key imported afterwards; z->z sends with change and memos;
// a t->z shielding send; spends of our notes and of someone else's. pwalletMain points at it during the test.
class HistoryWallet : public CWallet {
public:
bool EncryptKeys(CKeyingMaterial& vMasterKeyIn) { return CCryptoKeyStore::EncryptKeys(vMasterKeyIn); }
bool Unlock(const CKeyingMaterial& vMasterKeyIn) { return CCryptoKeyStore::Unlock(vMasterKeyIn); }
};
class WalletHistory : public ::testing::Test {
protected:
HistoryWallet wallet;
CWallet* savedWallet = nullptr;
SaplingPaymentAddress paA, paA2, paB, paC, paD, paF;
CTxDestination taddr;
std::map<uint256, int> depthOf;
std::vector<uint256> txids;
CCoinsView coinsBase;
CCoinsViewCache coins{&coinsBase};
CCoinsViewCache* savedCoinsTip = nullptr;
CBlockIndex* savedTip = nullptr;
std::deque<CBlock> blocks;
std::deque<CBlockIndex> index;
std::vector<uint256> insertedBlocks;
void SetUp() override {
SelectParams(CBaseChainParams::MAIN);
savedWallet = pwalletMain;
pwalletMain = &wallet;
savedCoinsTip = pcoinsTip;
pcoinsTip = &coins;
LOCK2(cs_main, wallet.cs_wallet);
savedTip = chainActive.Tip();
std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x2d);
ASSERT_TRUE(wallet.SetHDSeed(HDSeed(raw)));
HDSeed seed;
ASSERT_TRUE(wallet.GetHDSeedForDerivation(seed));
const uint256 seedOvk = ovkForShieldingFromTaddr(seed);
auto skA = Key(0), skB = Key(1), skC = Key(2), skD = Key(3), skF = Key(9);
paA = skA.DefaultAddress();
diversifier_index_t j;
j.begin()[0] = 50;
paA2 = skA.ToXFVK().Address(j)->second;
paB = skB.DefaultAddress();
paC = skC.DefaultAddress();
paD = skD.DefaultAddress();
paF = skF.DefaultAddress();
const uint256 ovkA = skA.expsk.full_viewing_key().ovk, ovkC = skC.expsk.full_viewing_key().ovk;
ASSERT_TRUE(wallet.AddSaplingZKey(skA, paA));
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(skA.expsk.full_viewing_key().in_viewing_key(), paA2));
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(skB.expsk.full_viewing_key().in_viewing_key(), paB));
ASSERT_TRUE(wallet.AddSaplingZKey(skC, paC));
CKey tkey;
tkey.MakeNewKey(true);
ASSERT_TRUE(wallet.AddKeyPubKey(tkey, tkey.GetPubKey()));
taddr = tkey.GetPubKey().GetID();
const uint256 nfA = uint256S("a1"), nfA2 = uint256S("a2"), nfC = uint256S("c1"), nfForeign = uint256S("ff");
// T0: transparent funds to our taddr.
CMutableTransaction t0;
t0.vin.push_back(CTxIn(COutPoint(GetRandHash(), 0)));
t0.vout.push_back(CTxOut(10 * COIN, GetScriptForDestination(taddr)));
AddTx(CTransaction(t0), 30);
// T1: received by A, A2, B (viewing key only), C and someone else; no spends.
uint256 t1 = AddTx(SaplingTx({Output(paA, 1000, "for A"), Output(paA2, 2000, "empty"), Output(paB, 3000, "for B"),
Output(paC, 4000, "binary"), Output(paF, 5000, "not ours")}, {}),
25, {{0, nfA}, {1, nfA2}, {3, nfC}});
(void)t1;
// T2: A sends to F with change back to A, spending A's note: outCiphertexts under A's ovk.
AddTx(SaplingTx({Output(paF, 600, "payment", ovkA), Output(paA, 399, "change", ovkA)}, {nfA}), 20);
// T3: t->z shielding of T0's output to A, outCiphertext under the seed's shielding ovk.
AddTx(SaplingTx({Output(paA, 9 * COIN, "empty", seedOvk)}, {}, {COutPoint(txids[0], 0)}), 15);
// T4: C spends its note: to A2 (another of our keys) and change to C.
AddTx(SaplingTx({Output(paA2, 1500, "to a diversified A", ovkC), Output(paC, 2499, "binary", ovkC)}, {nfC}), 10);
// T5: someone else's spend paying A.
AddTx(SaplingTx({Output(paA, 777, "from a stranger")}, {nfForeign}), 8);
// T6: received by D before D's key is in the wallet (an import without rescan), then mined.
AddTx(SaplingTx({Output(paD, 4242, "for D, found later")}, {}), 5);
// T8: spends one of our notes and one of someone else's (a coinjoin-like tx): missingSpendingKeys.
AddTx(SaplingTx({Output(paF, 100, "mixed", ovkA)}, {nfA, uint256S("fe")}), 3);
// T9: spends A2's note, but its one output's outCiphertext is under nobody's ovk we know: missingSaplingOVK.
AddTx(SaplingTx({Output(paF, 50, "unreadable send")}, {nfA2}), 2);
// T7: spends A2's note (A and A2 share A's key); never mined and not in the mempool (depth -1).
AddTx(SaplingTx({Output(paF, 2900, "", ovkA)}, {nfA2}), 0);
ASSERT_TRUE(wallet.AddSaplingZKey(skD, paD));
Mine();
}
void TearDown() override {
LOCK(cs_main);
chainActive.SetTip(savedTip);
for (const uint256& h : insertedBlocks)
mapBlockIndex.erase(h);
pcoinsTip = savedCoinsTip;
pwalletMain = savedWallet;
}
uint256 AddTx(const CTransaction& tx, int depth, const std::map<uint32_t, uint256>& nullifiers = {}) {
CWalletTx wtx(&wallet, tx);
mapSaplingNoteData_t found = wallet.FindMySaplingNotes(tx).first;
for (const auto& nf : nullifiers)
found.at(SaplingOutPoint(tx.GetHash(), nf.first)).nullifier = nf.second;
wtx.SetSaplingNoteData(found);
EXPECT_TRUE(wallet.AddToWallet(wtx, true, nullptr));
if (depth > 0)
depthOf[tx.GetHash()] = depth;
txids.push_back(tx.GetHash());
return tx.GetHash();
}
void Mine() {
const int tip = 40;
for (int h = 0; h <= tip; h++) {
blocks.emplace_back();
CBlock& block = blocks.back();
block.nTime = 1700000000 + 36 * h;
block.nNonce = uint256S(strprintf("%x", 0x900 + h));
if (h > 0)
block.hashPrevBlock = blocks[h - 1].GetHash();
for (const auto& d : depthOf)
if (tip - d.second + 1 == h)
block.vtx.push_back(wallet.mapWallet.at(d.first));
block.hashMerkleRoot = block.BuildMerkleTree();
index.emplace_back(block);
CBlockIndex& bi = index.back();
bi.SetHeight(h);
bi.pprev = h > 0 ? &index[h - 1] : nullptr;
auto inserted = mapBlockIndex.insert(std::make_pair(block.GetHash(), &bi));
ASSERT_TRUE(inserted.second);
insertedBlocks.push_back(block.GetHash());
bi.phashBlock = &inserted.first->first;
}
chainActive.SetTip(&index[tip]);
for (const auto& d : depthOf)
wallet.mapWallet.at(d.first).SetMerkleBranch(blocks[tip - d.second + 1]);
}
std::string NewJSON(const CWalletTx& wtx, const std::string& addr, bool fBool, int returnType) {
UniValue ret(UniValue::VARR);
zsWalletTxJSON(wtx, ret, addr, fBool, returnType, GetSaplingHistoryKeys());
return ret.write();
}
std::string LegacyJSON(const CWalletTx& wtx, const std::string& addr, bool fBool, int returnType) {
UniValue ret(UniValue::VARR);
Legacy_zsWalletTxJSON(wtx, ret, addr, fBool, returnType);
return ret.write();
}
void ExpectMatchesLegacy(bool checkCoverage = true);
};
// Every tx, every return type, unfiltered and filtered by each kind of address: byte-identical JSON. Run with
// the wallet's plaintext cache warm (as after the startup warm-up) and cold (every output trial-decrypted).
class WalletHistoryCold : public WalletHistory {};
TEST_F(WalletHistoryCold, MatchesLegacy) {
LOCK2(cs_main, wallet.cs_wallet);
for (auto& p : wallet.mapWallet)
for (auto& item : p.second.mapSaplingNoteData)
item.second.cachedPlaintext = boost::none;
ExpectMatchesLegacy();
}
TEST_F(WalletHistory, MatchesLegacy) {
LOCK2(cs_main, wallet.cs_wallet);
ExpectMatchesLegacy();
}
void WalletHistory::ExpectMatchesLegacy(bool checkCoverage) {
const std::vector<std::pair<std::string, bool>> filters = {
{"*", false}, {"*", true}, {EncodePaymentAddress(paA), true}, {EncodePaymentAddress(paA2), true},
{EncodePaymentAddress(paB), true}, {EncodePaymentAddress(paC), true}, {EncodePaymentAddress(paD), true},
{EncodePaymentAddress(paF), true}, {EncodeDestination(taddr), true},
};
std::string all;
for (const uint256& txid : txids) {
const CWalletTx& wtx = wallet.mapWallet.at(txid);
for (int returnType = 0; returnType <= 3; returnType++) {
for (const auto& f : filters) {
// New first: the legacy spends helper default-inserts into wallet maps.
std::string got = NewJSON(wtx, f.first, f.second, returnType);
std::string want = LegacyJSON(wtx, f.first, f.second, returnType);
EXPECT_EQ(want, got) << "tx " << txid.ToString() << " returnType " << returnType << " filter " << f.first << "/" << f.second;
all += got;
}
}
}
// The fixture really reaches the interesting branches.
if (!checkCoverage)
return;
for (const char* needle : {"\"change\":true", "\"memoStr\"", "\"saplingSpends\":[{", "\"transparentSpends\":[{",
"\"saplingSends\":[{", "\"saplingReceived\":[{", "\"missingSaplingOVK\":true",
"\"missingSpendingKeys\":true", "for D, found later"}) {
EXPECT_NE(std::string::npos, all.find(needle)) << needle;
}
}
// The same comparison on an encrypted wallet, locked (no spending keys or HD seed available: no shielded
// received or sent entries, as before) and then unlocked.
class WalletHistoryEncrypted : public WalletHistory {};
TEST_F(WalletHistoryEncrypted, MatchesLegacyLockedAndUnlocked) {
LOCK2(cs_main, wallet.cs_wallet);
CKeyingMaterial masterKey(32, 0x5e);
ASSERT_TRUE(wallet.EncryptKeys(masterKey));
ASSERT_TRUE(wallet.IsLocked());
ASSERT_TRUE(GetSaplingHistoryKeys().spendable.empty());
ExpectMatchesLegacy(false);
ASSERT_TRUE(wallet.Unlock(masterKey));
ASSERT_FALSE(GetSaplingHistoryKeys().spendable.empty());
ExpectMatchesLegacy();
}
// The history helpers only read the wallet. The old spends helper's operator[] lookups inserted an entry into
// mapSaplingNullifiersToNotes for every other wallet's nullifier they met, which made a later
// z_viewtransaction of that tx fail (map::at on a null outpoint).
TEST_F(WalletHistory, DoesNotModifyTheWallet) {
LOCK2(cs_main, wallet.cs_wallet);
const size_t nullifierEntries = wallet.mapSaplingNullifiersToNotes.size();
std::map<uint256, size_t> noteEntries;
for (const auto& p : wallet.mapWallet)
noteEntries[p.first] = p.second.mapSaplingNoteData.size();
for (const uint256& txid : txids)
for (int returnType = 0; returnType <= 3; returnType++)
NewJSON(wallet.mapWallet.at(txid), "*", false, returnType);
EXPECT_EQ(nullifierEntries, wallet.mapSaplingNullifiersToNotes.size());
EXPECT_EQ(0u, wallet.mapSaplingNullifiersToNotes.count(uint256S("ff"))) << "someone else's spend";
for (const auto& p : wallet.mapWallet)
EXPECT_EQ(noteEntries.at(p.first), p.second.mapSaplingNoteData.size());
// What the old helpers did, for the record.
for (const uint256& txid : txids)
LegacyJSON(wallet.mapWallet.at(txid), "*", false, 0);
EXPECT_EQ(1u, wallet.mapSaplingNullifiersToNotes.count(uint256S("ff")));
}
src/gtest/test_wallet_notecache.cpp
+1010
View File
File diff suppressed because it is too large Load Diff
src/gtest/test_wallet_notedata.cpp
+272
View File
@@ -0,0 +1,272 @@
// Copyright (c) 2016-2024 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
#include <gtest/gtest.h>
#include <deque>
#include "chain.h"
#include "coins.h"
#include "main.h"
#include "primitives/block.h"
#include "primitives/transaction.h"
#include "random.h"
#include "uint256.h"
#include "wallet/wallet.h"
#include "zcash/Note.hpp"
#include "zcash/zip32.h"
namespace {
class NoteDataWallet : public CWallet {
public:
bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock = true) {
return CWallet::UpdatedNoteData(wtxIn, wtx, fSameBlock);
}
int VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessOnly) {
return CWallet::VerifyAndSetInitialWitness(pindex, witnessOnly);
}
};
libzcash::SaplingIncomingViewingKey TestIvk()
{
return libzcash::SaplingIncomingViewingKey(uint256S("0101010101010101010101010101010101010101010101010101010101010101"));
}
// A note as the wallet holds it once witnessed: nullifier set, witnesses cached at a height.
SaplingNoteData WitnessedNote(const SaplingMerkleTree& tree, int height, const uint256& nullifier)
{
SaplingNoteData nd(TestIvk(), nullifier);
nd.witnesses.push_front(tree.witness());
nd.witnessHeight = height;
return nd;
}
// A note as FindMySaplingNotes returns it during a replay or rescan: no nullifier, no witnesses.
SaplingNoteData FreshNote()
{
return SaplingNoteData(TestIvk());
}
}
// A replayed or rescanned tx must keep the witnesses AND the nullifier it already had, and must not
// report an update: dropping the nullifier made the next block clear and reseed every carried witness.
TEST(WalletNoteData, ReplayKeepsNullifierAndWitnesses) {
NoteDataWallet wallet;
CWalletTx wtx(&wallet, CTransaction());
uint256 txid = uint256S("aa");
SaplingOutPoint op0(txid, 0);
SaplingMerkleTree tree;
tree.append(uint256S("01"));
const uint256 nf = uint256S("f00d");
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 100, nf);
wtx.mapSaplingNoteData[op0].witnessRootValidated = true;
auto before = wtx.mapSaplingNoteData;
CWalletTx replay(&wallet, CTransaction());
replay.mapSaplingNoteData[op0] = FreshNote();
EXPECT_FALSE(wallet.UpdatedNoteData(replay, wtx));
ASSERT_EQ(1u, wtx.mapSaplingNoteData.size());
const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
ASSERT_TRUE(static_cast<bool>(nd.nullifier));
EXPECT_EQ(nf, *nd.nullifier);
EXPECT_EQ(100, nd.witnessHeight);
EXPECT_TRUE(nd.witnesses == before.at(op0).witnesses);
// ...but the kept witness is re-checked against the active chain once (in-memory flag only).
EXPECT_FALSE(nd.witnessRootValidated);
}
// A tx re-mined in a different block (reorg) has its notes at new tree positions: the old nullifier
// must not be carried, so VerifyAndSetInitialWitness reseeds the note at its new position.
TEST(WalletNoteData, MovedTxDropsNullifier) {
NoteDataWallet wallet;
CWalletTx wtx(&wallet, CTransaction());
SaplingOutPoint op0(uint256S("ee"), 0);
SaplingMerkleTree tree;
tree.append(uint256S("04"));
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 500, uint256S("cafe"));
CWalletTx moved(&wallet, CTransaction());
moved.mapSaplingNoteData[op0] = FreshNote();
EXPECT_TRUE(wallet.UpdatedNoteData(moved, wtx, false));
const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
EXPECT_FALSE(static_cast<bool>(nd.nullifier));
}
// When the incoming data really adds something (a note for a newly added key), the tx is updated,
// and the notes it already had keep their witnesses, witness height and nullifier together.
TEST(WalletNoteData, NewNoteUpdatesAndKeepsExistingNoteState) {
NoteDataWallet wallet;
CWalletTx wtx(&wallet, CTransaction());
uint256 txid = uint256S("bb");
SaplingOutPoint op0(txid, 0);
SaplingOutPoint op1(txid, 1);
SaplingMerkleTree tree;
tree.append(uint256S("02"));
const uint256 nf = uint256S("beef");
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 200, nf);
CWalletTx incoming(&wallet, CTransaction());
incoming.mapSaplingNoteData[op0] = FreshNote();
incoming.mapSaplingNoteData[op1] = FreshNote();
EXPECT_TRUE(wallet.UpdatedNoteData(incoming, wtx));
ASSERT_EQ(2u, wtx.mapSaplingNoteData.size());
const SaplingNoteData& nd0 = wtx.mapSaplingNoteData.at(op0);
ASSERT_TRUE(static_cast<bool>(nd0.nullifier));
EXPECT_EQ(nf, *nd0.nullifier);
EXPECT_EQ(200, nd0.witnessHeight);
EXPECT_EQ(1u, nd0.witnesses.size());
const SaplingNoteData& nd1 = wtx.mapSaplingNoteData.at(op1);
EXPECT_FALSE(static_cast<bool>(nd1.nullifier));
EXPECT_TRUE(nd1.witnesses.empty());
EXPECT_EQ(-1, nd1.witnessHeight);
}
// Nothing is carried for a note that has no witnesses (PART B2): its height and nullifier describe
// no witness, so they must not survive onto the incoming data.
TEST(WalletNoteData, UnwitnessedNoteCarriesNothing) {
NoteDataWallet wallet;
CWalletTx wtx(&wallet, CTransaction());
SaplingOutPoint op0(uint256S("cc"), 0);
SaplingNoteData stale(TestIvk(), uint256S("dead"));
stale.witnessHeight = 300;
wtx.mapSaplingNoteData[op0] = stale;
CWalletTx incoming(&wallet, CTransaction());
incoming.mapSaplingNoteData[op0] = FreshNote();
EXPECT_TRUE(wallet.UpdatedNoteData(incoming, wtx));
const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
EXPECT_FALSE(static_cast<bool>(nd.nullifier));
EXPECT_EQ(-1, nd.witnessHeight);
EXPECT_TRUE(nd.witnesses.empty());
}
// Incoming data with no Sapling notes never touches the wallet's note data.
TEST(WalletNoteData, EmptyIncomingIsNoUpdate) {
NoteDataWallet wallet;
CWalletTx wtx(&wallet, CTransaction());
SaplingOutPoint op0(uint256S("dd"), 0);
SaplingMerkleTree tree;
tree.append(uint256S("03"));
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 400, uint256S("abcd"));
CWalletTx incoming(&wallet, CTransaction());
EXPECT_FALSE(wallet.UpdatedNoteData(incoming, wtx));
EXPECT_EQ(1u, wtx.mapSaplingNoteData.size());
EXPECT_EQ(400, wtx.mapSaplingNoteData.at(op0).witnessHeight);
}
namespace {
// A real encrypted Sapling output to pa (the wallet never checks its proof or value commitment).
OutputDescription EncryptedOutput(const libzcash::SaplingPaymentAddress& pa, uint64_t value)
{
libzcash::SaplingNote note(pa, value);
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
memo.fill(0);
auto enc = libzcash::SaplingNotePlaintext(note, memo).encrypt(note.pk_d);
assert(static_cast<bool>(enc));
OutputDescription od;
od.cv = GetRandHash();
od.cm = note.cm().get();
od.ephemeralKey = enc->second.get_epk();
od.encCiphertext = enc->first;
return od;
}
// A three-block active chain (heights 0-2) whose block 1 holds the given txs, undone on destruction:
// chainActive, mapBlockIndex and pcoinsTip are process-wide and every gtest suite shares the process.
struct FakeActiveChain {
std::deque<CBlock> blocks;
std::deque<CBlockIndex> index;
CBlockIndex* savedTip;
CCoinsView coinsBase;
CCoinsViewCache coins{&coinsBase};
CCoinsViewCache* savedCoinsTip;
explicit FakeActiveChain(const std::vector<CTransaction>& inBlock1) {
LOCK(cs_main);
savedTip = chainActive.Tip();
savedCoinsTip = pcoinsTip;
pcoinsTip = &coins;
for (int h = 0; h <= 2; h++) {
blocks.emplace_back();
CBlock& block = blocks.back();
block.nTime = 1700000000 + h;
block.nNonce = uint256S(strprintf("%x", 0x700 + h));
if (h > 0)
block.hashPrevBlock = blocks[h - 1].GetHash();
if (h == 1)
block.vtx = inBlock1;
block.hashMerkleRoot = block.BuildMerkleTree();
index.emplace_back(block);
CBlockIndex& bi = index.back();
bi.SetHeight(h);
bi.pprev = h > 0 ? &index[h - 1] : nullptr;
bi.phashBlock = &mapBlockIndex.insert(std::make_pair(block.GetHash(), &bi)).first->first;
}
chainActive.SetTip(&index.back());
}
~FakeActiveChain() {
LOCK(cs_main);
chainActive.SetTip(savedTip);
for (const CBlock& block : blocks)
mapBlockIndex.erase(block.GetHash());
pcoinsTip = savedCoinsTip;
}
};
}
// A note found with an incoming viewing key only (z_importviewingkey) can't be spent and never gets
// a nullifier, because deriving one takes the full viewing key. VerifyAndSetInitialWitness must leave
// it alone. It used to clear it and reseed its witness from the block on disk on every call, then
// abort on the missing nullifier the first time the note was confirmed. Here there are no block
// files, so the old reseed path throws.
TEST(WalletNoteData, ViewingKeyOnlyNoteIsNotWitnessed) {
NoteDataWallet wallet;
std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x3c);
HDSeed seed(raw);
auto sk = libzcash::SaplingExtendedSpendingKey::Master(seed);
auto pa = sk.DefaultAddress();
LOCK2(cs_main, wallet.cs_wallet);
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(sk.expsk.full_viewing_key().in_viewing_key(), pa));
ASSERT_FALSE(wallet.HaveSaplingFullViewingKey(sk.expsk.full_viewing_key().in_viewing_key()));
CMutableTransaction mtx;
mtx.fOverwintered = true;
mtx.nVersion = SAPLING_TX_VERSION;
mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
mtx.vShieldedOutput.push_back(EncryptedOutput(pa, 5000));
CTransaction tx(mtx);
CWalletTx wtx(&wallet, tx);
mapSaplingNoteData_t found = wallet.FindMySaplingNotes(tx).first;
ASSERT_EQ(1u, found.size()) << "the viewing key finds the note";
wtx.SetSaplingNoteData(found);
ASSERT_TRUE(wallet.AddToWallet(wtx, true, nullptr));
FakeActiveChain chain({tx});
CWalletTx& stored = wallet.mapWallet.at(tx.GetHash());
stored.SetMerkleBranch(chain.blocks[1]);
ASSERT_EQ(2, stored.GetDepthInMainChain());
int minHeight = -1;
ASSERT_NO_THROW(minHeight = wallet.VerifyAndSetInitialWitness(chainActive.Tip(), false));
EXPECT_EQ(chainActive.Height(), minHeight) << "it must not pull the witness rebuild down to its height";
const SaplingNoteData& nd = stored.mapSaplingNoteData.at(SaplingOutPoint(tx.GetHash(), 0));
EXPECT_TRUE(nd.witnesses.empty());
EXPECT_FALSE(static_cast<bool>(nd.nullifier));
EXPECT_EQ(-1, nd.witnessHeight);
}
src/gtest/test_wallet_scan_notes.cpp
-848
View File
@@ -1,848 +0,0 @@
// Copyright (c) 2016-2026 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
//
// Proves that the optimised CWallet::GetFilteredNotes returns exactly what the original
// did: the same entries, in the same order, with every field equal. The optimisations
// are references instead of copies, skipping txs without note data, the spent check
// before trial decryption, the exact ivk pre-filter for address-filtered calls, and
// building the note and address from decrypt_with_pk_d. RefGetFilteredNotes below is a
// verbatim copy of the original body (c1040028e), and both run over the same hand-built
// wallet for a grid of filters and parameters. The file also tests
// SaplingNotePlaintext::decrypt_with_pk_d against decrypt().
//
// No proofs are needed: the wallet never verifies them, so notes are encrypted by hand
// into fabricated OutputDescriptions and added with AddToWallet(wtx, true, NULL). Depth
// is faked with a mapBlockIndex chain (depth >= 1), the mempool (depth 0) or neither
// (depth -1). minDepth > 1 is not covered: that branch calls tx_height(), which reads
// the tx from disk (the regtest differential covers it).
#include <gtest/gtest.h>
#include "arith_uint256.h"
#include "chain.h"
#include "chainparams.h"
#include "key_io.h"
#include "main.h"
#include "primitives/transaction.h"
#include "txmempool.h"
#include "uint256.h"
#include "util.h"
#include "wallet/crypter.h"
#include "wallet/wallet.h"
#include "zcash/Address.hpp"
#include "zcash/Note.hpp"
#include "zcash/zip32.h"
#include <algorithm>
#include <array>
#include <cassert>
#include <climits>
#include <list>
#include <memory>
#include <set>
#include <string>
#include <vector>
#include <boost/filesystem.hpp>
// The original body calls these; wallet.cpp declares them the same way (wallet.cpp:77-78).
// Both are defined in main.cpp (via hush.h). Neither is reached here (minDepth <= 1).
int32_t hush_dpowconfs(int32_t height,int32_t numconfs);
int tx_height( const uint256 &hash );
/**
* Verbatim copy of the original
* CWallet::GetFilteredNotes(std::vector<SaplingNoteEntry>&, std::set<PaymentAddress>&,
* int, int, bool, bool, bool) at c1040028e (wallet.cpp:5637-5721). The only changes:
* it is a free function, so wallet members are `wallet.`-qualified (all of them are
* public: cs_wallet, mapWallet, IsSaplingSpent, GetSaplingIncomingViewingKey,
* GetSaplingFullViewingKey, HaveSaplingSpendingKey, IsLockedNote), and the two LogPrintf
* calls are dropped (they do not affect the output).
*/
static void RefGetFilteredNotes(
CWallet& wallet,
std::vector<SaplingNoteEntry>& saplingEntries,
std::set<libzcash::PaymentAddress>& filterAddresses,
int minDepth,
int maxDepth,
bool ignoreSpent,
bool requireSpendingKey,
bool ignoreLocked)
{
using namespace libzcash;
LOCK2(cs_main, wallet.cs_wallet);
for (auto & p : wallet.mapWallet) {
CWalletTx wtx = p.second;
// Filter the transactions before checking for notes
if (!CheckFinalTx(wtx) || wtx.GetBlocksToMaturity() > 0)
continue;
if (minDepth > 1) {
int nHeight = tx_height(wtx.GetHash());
if ( nHeight == 0 ) {
continue;
}
int nDepth = wtx.GetDepthInMainChain();
int dpowconfs = hush_dpowconfs(nHeight,nDepth);
if ( dpowconfs < minDepth || dpowconfs > maxDepth) {
continue;
}
} else {
if ( wtx.GetDepthInMainChain() < minDepth ||
wtx.GetDepthInMainChain() > maxDepth) {
continue;
}
}
for (auto & pair : wtx.mapSaplingNoteData) {
SaplingOutPoint op = pair.first;
SaplingNoteData nd = pair.second;
auto maybe_pt = SaplingNotePlaintext::decrypt(
wtx.vShieldedOutput[op.n].encCiphertext,
nd.ivk,
wtx.vShieldedOutput[op.n].ephemeralKey,
wtx.vShieldedOutput[op.n].cm);
assert(static_cast<bool>(maybe_pt));
auto notePt = maybe_pt.get();
auto maybe_pa = nd.ivk.address(notePt.d);
assert(static_cast<bool>(maybe_pa));
auto pa = maybe_pa.get();
// skip notes which belong to a different payment address in the wallet
if (!(filterAddresses.empty() || filterAddresses.count(pa))) {
continue;
}
if (ignoreSpent && nd.nullifier && wallet.IsSaplingSpent(*nd.nullifier)) {
continue;
}
// skip notes which cannot be spent
if (requireSpendingKey) {
libzcash::SaplingIncomingViewingKey ivk;
libzcash::SaplingFullViewingKey fvk;
if (!(wallet.GetSaplingIncomingViewingKey(pa, ivk) &&
wallet.GetSaplingFullViewingKey(ivk, fvk) &&
wallet.HaveSaplingSpendingKey(fvk))) {
continue;
}
}
// skip locked notes
if (ignoreLocked && wallet.IsLockedNote(op)) {
continue;
}
auto note = notePt.note(nd.ivk).get();
saplingEntries.push_back(SaplingNoteEntry {
op, pa, note, notePt.memo(), wtx.GetDepthInMainChain() });
}
}
}
namespace {
// Heights 0..CHAIN_LENGTH-1, so a tx in the block at height h has depth CHAIN_LENGTH - h.
const int CHAIN_LENGTH = 10;
// The Jubjub subgroup order r_J. An ivk + r_J generates the same addresses as ivk,
// because librustzcash_ivk_to_pkd multiplies by the raw 256-bit value.
const char JUBJUB_R_HEX[] = "0e7db4ea6533afa906673b0101343b00a6682093ccc81082d0970e5ed6f72cb7";
// Keys and addresses of the test wallet (see WalletScanNotes::BuildWallet).
struct TestKeys {
libzcash::SaplingFullViewingKey fvk1, fvk2;
libzcash::SaplingIncomingViewingKey ivk1, ivk2, ivk3, ivk4, ivkN, aliasIvk1;
libzcash::SaplingPaymentAddress a1; // K1 default address (spending key)
libzcash::SaplingPaymentAddress d1; // K1 diversified address, recorded correctly
libzcash::SaplingPaymentAddress u1; // K1 diversified address, never recorded
libzcash::SaplingPaymentAddress x1; // K1 diversified address, recorded with K2's ivk
libzcash::SaplingPaymentAddress y1; // K1 diversified address, recorded with ivk1 + r_J
libzcash::SaplingPaymentAddress a2; // K2 default address (spending key)
libzcash::SaplingPaymentAddress a3; // K3: incoming viewing key only
libzcash::SaplingPaymentAddress a4; // K4: full viewing key only
libzcash::SaplingPaymentAddress aN; // address of a non-canonical ivk (>= 2^251)
libzcash::SaplingPaymentAddress foreign; // not in the wallet at all
std::vector<libzcash::SaplingPaymentAddress> empty; // spending keys that hold no notes
size_t nNotes;
};
// One output of a fabricated tx that the wallet owns: the address it pays, the ivk the
// wallet records for it (SaplingNoteData::ivk) and its cached nullifier, if any.
struct OwnedOutput {
libzcash::SaplingPaymentAddress addr;
libzcash::SaplingIncomingViewingKey ivk;
uint64_t value;
boost::optional<uint256> nullifier;
};
struct NamedFilter {
std::string name;
std::set<libzcash::PaymentAddress> addrs;
};
// CCryptoKeyStore::EncryptKeys is protected, and CWallet::EncryptWallet needs a wallet file.
class EncryptableWallet : public CWallet {
public:
bool EncryptKeysForTest(CKeyingMaterial& vMasterKeyIn) { return EncryptKeys(vMasterKeyIn); }
};
libzcash::SaplingExtendedSpendingKey KeyFromSeedByte(unsigned char b)
{
RawHDSeed raw(32, b);
HDSeed seed(raw);
return libzcash::SaplingExtendedSpendingKey::Master(seed);
}
// A diversified address of xsk other than its default one: the first valid diversifier
// index at or after `start` (the default address uses the first one from 0).
libzcash::SaplingPaymentAddress DiversifiedAddress(
const libzcash::SaplingExtendedSpendingKey& xsk, unsigned char start)
{
libzcash::diversifier_index_t j;
j.begin()[0] = start;
return xsk.ToXFVK().Address(j).get().second;
}
// A synthetic nullifier. GetFilteredNotes only compares nullifiers, it never recomputes them.
uint256 Nf(unsigned char n)
{
uint256 nf;
nf.begin()[0] = n;
nf.begin()[31] = 0x4e;
return nf;
}
// A shielded output paying `value` to `addr`, encrypted the way a real sender does
// (SaplingNotePlaintext::encrypt), with a random rcm and a memo derived from the value.
// There is no proof, value commitment or outgoing ciphertext: the wallet reads only cm,
// ephemeralKey and encCiphertext.
OutputDescription MakeOutput(const libzcash::SaplingPaymentAddress& addr, uint64_t value)
{
libzcash::SaplingNote note(addr, value);
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
memo.fill(0);
memo[0] = value & 0xff;
memo[1] = (value >> 8) & 0xff;
libzcash::SaplingNotePlaintext pt(note, memo);
libzcash::SaplingNotePlaintextEncryptionResult res = pt.encrypt(note.pk_d).get();
OutputDescription od;
od.cv = uint256();
od.cm = note.cm().get();
od.ephemeralKey = res.second.get_epk();
od.encCiphertext = res.first;
od.outCiphertext.fill(0);
od.zkproof.fill(0);
return od;
}
std::set<libzcash::PaymentAddress> Filter(const std::vector<libzcash::SaplingPaymentAddress>& addrs)
{
std::set<libzcash::PaymentAddress> s;
for (const libzcash::SaplingPaymentAddress& a : addrs) {
s.insert(a);
}
return s;
}
void ExpectSameEntries(const std::vector<SaplingNoteEntry>& expected,
const std::vector<SaplingNoteEntry>& actual)
{
ASSERT_EQ(expected.size(), actual.size());
for (size_t i = 0; i < expected.size(); i++) {
SCOPED_TRACE(testing::Message() << "entry " << i);
const SaplingNoteEntry& e = expected[i];
const SaplingNoteEntry& a = actual[i];
EXPECT_EQ(e.op.hash, a.op.hash);
EXPECT_EQ(e.op.n, a.op.n);
EXPECT_TRUE(e.address == a.address);
EXPECT_TRUE(e.note.d == a.note.d);
EXPECT_EQ(e.note.pk_d, a.note.pk_d);
EXPECT_EQ(e.note.value(), a.note.value());
EXPECT_EQ(e.note.r, a.note.r);
EXPECT_TRUE(e.memo == a.memo);
EXPECT_EQ(e.confirmations, a.confirmations);
}
}
// Runs the original and the optimised GetFilteredNotes with the same arguments and
// expects the same entries in the same order. Returns the number of entries.
size_t ExpectMatchesReference(CWallet& wallet, NamedFilter& filter,
int minDepth, int maxDepth,
bool ignoreSpent, bool requireSpendingKey, bool ignoreLocked)
{
SCOPED_TRACE(testing::Message() << "filter=" << filter.name
<< " minDepth=" << minDepth << " maxDepth=" << maxDepth
<< " ignoreSpent=" << ignoreSpent
<< " requireSpendingKey=" << requireSpendingKey
<< " ignoreLocked=" << ignoreLocked);
std::vector<SaplingNoteEntry> expected;
std::vector<SaplingNoteEntry> actual;
RefGetFilteredNotes(wallet, expected, filter.addrs,
minDepth, maxDepth, ignoreSpent, requireSpendingKey, ignoreLocked);
wallet.GetFilteredNotes(actual, filter.addrs,
minDepth, maxDepth, ignoreSpent, requireSpendingKey, ignoreLocked);
ExpectSameEntries(expected, actual);
return expected.size();
}
// The reference's result size with maxDepth INT_MAX.
size_t RefCount(CWallet& wallet, std::set<libzcash::PaymentAddress> filter, int minDepth,
bool ignoreSpent, bool requireSpendingKey, bool ignoreLocked)
{
std::vector<SaplingNoteEntry> entries;
RefGetFilteredNotes(wallet, entries, filter,
minDepth, INT_MAX, ignoreSpent, requireSpendingKey, ignoreLocked);
return entries.size();
}
// Every filter shape the pre-filter distinguishes.
std::vector<NamedFilter> AllFilters(const TestKeys& k)
{
// 16 recorded, verifiable addresses: the pre-filter is on (the cap is inclusive).
std::vector<libzcash::SaplingPaymentAddress> sixteen = {k.a1, k.d1, k.a2, k.a3, k.a4};
sixteen.insert(sixteen.end(), k.empty.begin(), k.empty.begin() + 11);
// 17: the pre-filter is off only because of the cap.
std::vector<libzcash::SaplingPaymentAddress> seventeen = sixteen;
seventeen.push_back(k.empty[11]);
std::set<libzcash::PaymentAddress> invalidOnly;
invalidOnly.insert(libzcash::InvalidEncoding());
std::set<libzcash::PaymentAddress> invalidAndA2 = invalidOnly;
invalidAndA2.insert(k.a2);
std::vector<NamedFilter> f;
f.push_back(NamedFilter{"{}", Filter({})});
f.push_back(NamedFilter{"{a1}", Filter({k.a1})});
f.push_back(NamedFilter{"{d1}", Filter({k.d1})});
f.push_back(NamedFilter{"{a2}", Filter({k.a2})});
f.push_back(NamedFilter{"{a3 ivk-only}", Filter({k.a3})});
f.push_back(NamedFilter{"{a4 fvk-only}", Filter({k.a4})});
f.push_back(NamedFilter{"{a1,a2,a3,a4,d1}", Filter({k.a1, k.a2, k.a3, k.a4, k.d1})});
f.push_back(NamedFilter{"{u1 unrecorded}", Filter({k.u1})});
f.push_back(NamedFilter{"{a1,u1 unrecorded}", Filter({k.a1, k.u1})});
f.push_back(NamedFilter{"{x1 mismatched}", Filter({k.x1})});
f.push_back(NamedFilter{"{x1 mismatched,a2}", Filter({k.x1, k.a2})});
f.push_back(NamedFilter{"{y1 aliased}", Filter({k.y1})});
f.push_back(NamedFilter{"{aN non-canonical}", Filter({k.aN})});
f.push_back(NamedFilter{"{no notes}", Filter({k.empty[0]})});
f.push_back(NamedFilter{"{foreign}", Filter({k.foreign})});
f.push_back(NamedFilter{"{invalid}", invalidOnly});
f.push_back(NamedFilter{"{invalid,a2}", invalidAndA2});
f.push_back(NamedFilter{"16 addresses", Filter(sixteen)});
f.push_back(NamedFilter{"17 addresses", Filter(seventeen)});
return f;
}
// decrypt_with_pk_d must succeed exactly when decrypt() does, and must leave pk_d_out
// alone when it fails. Returns whether decryption succeeded.
bool DecryptBothWays(const libzcash::SaplingEncCiphertext& ct, const uint256& ivk,
const uint256& epk, const uint256& cmu)
{
uint256 sentinel;
std::fill(sentinel.begin(), sentinel.end(), 0xa5);
uint256 pk_d = sentinel;
auto expected = libzcash::SaplingNotePlaintext::decrypt(ct, ivk, epk, cmu);
auto actual = libzcash::SaplingNotePlaintext::decrypt_with_pk_d(ct, ivk, epk, cmu, pk_d);
EXPECT_EQ(static_cast<bool>(expected), static_cast<bool>(actual));
if (!actual) {
EXPECT_EQ(pk_d, sentinel);
}
return static_cast<bool>(actual);
}
} // namespace
class WalletScanNotes : public ::testing::Test {
protected:
boost::filesystem::path datadir;
bool hadDatadirArg;
std::string oldDatadirArg;
bool oldPrintToDebugLog;
CBlockIndex* oldTip;
std::vector<std::unique_ptr<CBlockIndex>> blocks;
std::vector<CTransaction> mempoolTxs;
void SetUp() override
{
SelectParams(CBaseChainParams::REGTEST);
// GetFilteredNotes logs on every call: keep that away from any real datadir and
// out of debug.log.
hadDatadirArg = mapArgs.count("-datadir") > 0;
if (hadDatadirArg) {
oldDatadirArg = mapArgs["-datadir"];
}
datadir = boost::filesystem::temp_directory_path() / boost::filesystem::unique_path();
boost::filesystem::create_directories(datadir);
mapArgs["-datadir"] = datadir.string();
ClearDatadirCache();
oldPrintToDebugLog = fPrintToDebugLog;
fPrintToDebugLog = false;
// A fake active chain. Only what CMerkleTx::GetDepthInMainChain reads is set: the
// mapBlockIndex entry, phashBlock, the height and pprev.
LOCK(cs_main);
oldTip = chainActive.Tip();
for (int h = 0; h < CHAIN_LENGTH; h++) {
std::unique_ptr<CBlockIndex> pindex(new CBlockIndex());
pindex->SetHeight(h);
pindex->pprev = blocks.empty() ? NULL : blocks.back().get();
uint256 hash;
hash.begin()[0] = h;
hash.begin()[30] = 0x5c;
hash.begin()[31] = 0xa9;
auto ret = mapBlockIndex.insert(std::make_pair(hash, pindex.get()));
ASSERT_TRUE(ret.second);
pindex->phashBlock = &ret.first->first;
blocks.push_back(std::move(pindex));
}
chainActive.SetTip(blocks.back().get());
}
void TearDown() override
{
{
LOCK(cs_main);
for (const CTransaction& tx : mempoolTxs) {
std::list<CTransaction> removed;
mempool.remove(tx, removed, false);
}
mempoolTxs.clear();
chainActive.SetTip(oldTip);
for (const std::unique_ptr<CBlockIndex>& pindex : blocks) {
mapBlockIndex.erase(pindex->GetBlockHash());
}
blocks.clear();
}
fPrintToDebugLog = oldPrintToDebugLog;
if (hadDatadirArg) {
mapArgs["-datadir"] = oldDatadirArg;
} else {
mapArgs.erase("-datadir");
}
ClearDatadirCache();
boost::system::error_code ec;
boost::filesystem::remove_all(datadir, ec);
// The network every other test in this binary selects for itself.
SelectParams(CBaseChainParams::MAIN);
}
// Adds a fabricated tx to the wallet the way LoadWallet does: AddToWallet(wtx, true,
// NULL) skips the database and proof checks but fills the nullifier and spend maps.
// `owned` become outputs 0..n-1 with wallet note data, `foreign` outputs follow with
// none, and `spends` are the nullifiers the tx reveals. depth >= 1 puts the tx in the
// fake chain, 0 in the mempool, -1 nowhere (like a conflicted tx).
uint256 AddTx(CWallet& wallet, int depth,
const std::vector<OwnedOutput>& owned,
const std::vector<libzcash::SaplingPaymentAddress>& foreign,
const std::vector<uint256>& spends,
bool transparentOutput = false)
{
assert(depth >= -1 && depth <= CHAIN_LENGTH);
CMutableTransaction mtx;
mtx.fOverwintered = true;
mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
mtx.nVersion = SAPLING_TX_VERSION;
for (const OwnedOutput& o : owned) {
mtx.vShieldedOutput.push_back(MakeOutput(o.addr, o.value));
}
for (const libzcash::SaplingPaymentAddress& addr : foreign) {
mtx.vShieldedOutput.push_back(MakeOutput(addr, 1));
}
for (const uint256& nf : spends) {
SpendDescription sd;
sd.nullifier = nf;
sd.zkproof.fill(0);
sd.spendAuthSig.fill(0);
mtx.vShieldedSpend.push_back(sd);
}
if (transparentOutput) {
mtx.vout.push_back(CTxOut(1000, CScript() << OP_TRUE));
}
CTransaction tx(mtx);
CWalletTx wtx(&wallet, tx);
for (size_t i = 0; i < owned.size(); i++) {
SaplingNoteData nd(owned[i].ivk);
nd.nullifier = owned[i].nullifier;
wtx.mapSaplingNoteData[SaplingOutPoint(tx.GetHash(), i)] = nd;
}
if (depth >= 1) {
wtx.hashBlock = blocks[CHAIN_LENGTH - depth]->GetBlockHash();
wtx.nIndex = 0;
// There is no real block to check a merkle branch against.
wtx.fMerkleVerified = true;
} else if (depth == 0) {
mempool.addUnchecked(tx.GetHash(),
CTxMemPoolEntry(tx, 0, GetTime(), 0.0, chainActive.Height(), true, false, SAPLING_BRANCH_ID),
false);
mempoolTxs.push_back(tx);
}
wallet.AddToWallet(wtx, true, NULL);
return tx.GetHash();
}
// The test wallet. Every key state, every recorded-address state and every note state
// GetFilteredNotes distinguishes, spread over several depths:
// K1 spending key: a1 (default), d1 (recorded diversified), u1 (unrecorded),
// x1 (recorded with K2's ivk, as z_importviewingkey allows), y1 (recorded with
// ivk1 + r_J, which still generates y1);
// K2 spending key: a2; K3 ivk only: a3; K4 fvk only: a4;
// ivkN: a non-canonical ivk in [2^251, r_J) and its address aN;
// 12 more spending keys with no notes; one foreign key.
// Notes are spent in a block, spent in the mempool, "spent" by a conflicted tx (so
// still unspent), unspent, without a cached nullifier although spent on chain, and
// locked. There are also txs without note data (spends only, transparent only, and
// a shielded output to a foreign address).
void BuildWallet(CWallet& wallet, TestKeys& k)
{
LOCK2(cs_main, wallet.cs_wallet);
libzcash::SaplingExtendedSpendingKey xsk1 = KeyFromSeedByte(0x11);
libzcash::SaplingExtendedSpendingKey xsk2 = KeyFromSeedByte(0x22);
libzcash::SaplingExtendedSpendingKey xsk3 = KeyFromSeedByte(0x33);
libzcash::SaplingExtendedSpendingKey xsk4 = KeyFromSeedByte(0x44);
libzcash::SaplingExtendedSpendingKey xskN = KeyFromSeedByte(0x55);
k.fvk1 = xsk1.expsk.full_viewing_key();
k.fvk2 = xsk2.expsk.full_viewing_key();
libzcash::SaplingFullViewingKey fvk4 = xsk4.expsk.full_viewing_key();
k.ivk1 = k.fvk1.in_viewing_key();
k.ivk2 = k.fvk2.in_viewing_key();
k.ivk3 = xsk3.expsk.full_viewing_key().in_viewing_key();
k.ivk4 = fvk4.in_viewing_key();
k.a1 = xsk1.DefaultAddress();
k.d1 = DiversifiedAddress(xsk1, 50);
k.u1 = DiversifiedAddress(xsk1, 100);
k.x1 = DiversifiedAddress(xsk1, 150);
k.y1 = DiversifiedAddress(xsk1, 200);
std::set<libzcash::SaplingPaymentAddress> k1addrs = {k.a1, k.d1, k.u1, k.x1, k.y1};
ASSERT_EQ(k1addrs.size(), (size_t)5);
k.a2 = xsk2.DefaultAddress();
k.a3 = xsk3.DefaultAddress();
k.a4 = xsk4.DefaultAddress();
k.foreign = KeyFromSeedByte(0x66).DefaultAddress();
// Genuine ivks are canonical (top 5 bits clear, so below 2^251).
ASSERT_EQ(k.ivk1.begin()[31] & 0xF8, 0);
ASSERT_EQ(k.ivk2.begin()[31] & 0xF8, 0);
// ivk1 + r_J generates exactly ivk1's addresses. If this ever stops holding, the
// y1 case no longer exercises the pre-filter's canonical-ivk guard.
k.aliasIvk1 = libzcash::SaplingIncomingViewingKey(ArithToUint256(
UintToArith256(k.ivk1) + UintToArith256(uint256S(JUBJUB_R_HEX))));
ASSERT_NE(k.aliasIvk1.begin()[31] & 0xF8, 0);
auto aliasY1 = k.aliasIvk1.address(k.y1.d);
ASSERT_TRUE(static_cast<bool>(aliasY1));
EXPECT_TRUE(aliasY1.get() == k.y1);
// ivkN in [2^251, 2^251 + 2^248), below r_J: ka_agree accepts it, so notes to aN
// decrypt normally, but it is not canonical.
uint256 rawN = xskN.expsk.full_viewing_key().in_viewing_key();
rawN.begin()[31] = 0x08;
k.ivkN = libzcash::SaplingIncomingViewingKey(rawN);
k.aN = k.ivkN.address(k.a1.d).get();
ASSERT_TRUE(wallet.AddSaplingZKey(xsk1, k.a1));
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(k.ivk1, k.d1));
ASSERT_TRUE(wallet.AddSaplingZKey(xsk2, k.a2));
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(k.ivk3, k.a3));
ASSERT_TRUE(wallet.AddSaplingFullViewingKey(fvk4, k.a4));
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(k.ivk2, k.x1));
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(k.aliasIvk1, k.y1));
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(k.ivkN, k.aN));
for (int i = 0; i < 12; i++) {
libzcash::SaplingExtendedSpendingKey xsk = KeyFromSeedByte(0x70 + i);
libzcash::SaplingPaymentAddress addr = xsk.DefaultAddress();
ASSERT_TRUE(wallet.AddSaplingZKey(xsk, addr));
k.empty.push_back(addr);
}
const boost::optional<uint256> noNf;
std::vector<uint256> noSpends;
std::vector<libzcash::SaplingPaymentAddress> noForeign;
// Mixed ownership in one tx; Nf(2) is spent in a block.
AddTx(wallet, 5, {{k.a1, k.ivk1, 1001, Nf(1)},
{k.a1, k.ivk1, 1002, Nf(2)},
{k.a2, k.ivk2, 2001, Nf(3)}}, noForeign, noSpends);
// Nf(4) is spent in the mempool. The IVK-only note has no cached nullifier, but its
// on-chain nullifier Nf(5) is spent in a block: it must still count as unspent.
AddTx(wallet, 1, {{k.d1, k.ivk1, 1003, Nf(4)},
{k.a3, k.ivk3, 3001, noNf}}, noForeign, noSpends);
// u1's note is locked; x1's note is "spent" only by a conflicted tx.
uint256 hashUX = AddTx(wallet, 3, {{k.u1, k.ivk1, 1004, Nf(6)},
{k.x1, k.ivk1, 1005, Nf(7)}}, noForeign, noSpends);
// In the mempool (depth 0).
AddTx(wallet, 0, {{k.a1, k.ivk1, 1006, Nf(8)},
{k.a4, k.ivk4, 4001, Nf(9)}}, noForeign, noSpends);
// Conflicted (depth -1).
AddTx(wallet, -1, {{k.a2, k.ivk2, 2002, Nf(10)}}, noForeign, noSpends);
// Deepest block; Nf(11) is spent in a block.
AddTx(wallet, 10, {{k.y1, k.ivk1, 1007, Nf(11)},
{k.aN, k.ivkN, 5001, noNf}}, noForeign, noSpends);
// a1's note is locked; one output goes to a foreign address (no note data).
uint256 hashLocked = AddTx(wallet, 2, {{k.a1, k.ivk1, 1008, Nf(12)},
{k.a2, k.ivk2, 2003, Nf(13)}},
{k.foreign}, noSpends);
// Nf(14) is spent in a block.
AddTx(wallet, 4, {{k.a3, k.ivk3, 3002, noNf},
{k.aN, k.ivkN, 5002, Nf(14)},
{k.x1, k.ivk1, 1009, Nf(15)}}, noForeign, noSpends);
AddTx(wallet, 1, {{k.d1, k.ivk1, 1010, Nf(16)}}, noForeign, noSpends);
k.nNotes = 18;
// Txs without note data: the spenders (in a block, in the mempool, conflicted), a
// transparent-only tx and a tx whose only output pays a foreign address.
AddTx(wallet, 2, {}, noForeign, {Nf(2), Nf(5), Nf(11), Nf(14)});
AddTx(wallet, 0, {}, noForeign, {Nf(4)});
AddTx(wallet, -1, {}, noForeign, {Nf(7)});
AddTx(wallet, 3, {}, noForeign, noSpends, true);
AddTx(wallet, 1, {}, {k.foreign}, noSpends);
wallet.LockNote(SaplingOutPoint(hashUX, 0));
wallet.LockNote(SaplingOutPoint(hashLocked, 0));
}
};
// The fixture really has what the equivalence tests rely on: every note is visible, the
// spent, depth, key and lock states are the intended ones, and the regression cases
// (unrecorded, mismatched, aliased and non-canonical addresses) hold notes, so equivalence
// on them is not vacuous.
TEST_F(WalletScanNotes, ReferenceSeesTheFixture)
{
CWallet wallet;
TestKeys k;
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
std::set<libzcash::PaymentAddress> none;
std::vector<SaplingNoteEntry> all;
RefGetFilteredNotes(wallet, all, none, -1, INT_MAX, false, false, false);
EXPECT_EQ(all.size(), k.nNotes);
std::set<int> depths;
for (const SaplingNoteEntry& e : all) {
depths.insert(e.confirmations);
}
EXPECT_EQ(depths, (std::set<int>{-1, 0, 1, 2, 3, 4, 5, 10}));
// Spent: Nf(2), Nf(11) and Nf(14) in a block, Nf(4) in the mempool. Not spent: Nf(7)
// (conflicted spender) and the IVK-only note without a cached nullifier.
EXPECT_EQ(RefCount(wallet, none, -1, true, false, false), k.nNotes - 4);
// minDepth 0 drops the conflicted tx (1 note), minDepth 1 also the mempool tx (2).
EXPECT_EQ(RefCount(wallet, none, 0, false, false, false), k.nNotes - 1);
EXPECT_EQ(RefCount(wallet, none, 1, false, false, false), k.nNotes - 3);
EXPECT_EQ(RefCount(wallet, none, -1, false, false, true), k.nNotes - 2);
// Spending key reachable through the recorded address: a1 (4), d1 (2), a2 (3), and
// x1 (2), because x1 is recorded with K2's ivk and the wallet has K2's spending key.
EXPECT_EQ(RefCount(wallet, none, -1, false, true, false), (size_t)11);
EXPECT_EQ(RefCount(wallet, Filter({k.a1}), -1, false, false, false), (size_t)4);
EXPECT_EQ(RefCount(wallet, Filter({k.d1}), -1, false, false, false), (size_t)2);
EXPECT_EQ(RefCount(wallet, Filter({k.u1}), -1, false, false, false), (size_t)1);
EXPECT_EQ(RefCount(wallet, Filter({k.x1}), -1, false, false, false), (size_t)2);
EXPECT_EQ(RefCount(wallet, Filter({k.y1}), -1, false, false, false), (size_t)1);
EXPECT_EQ(RefCount(wallet, Filter({k.a2}), -1, false, false, false), (size_t)3);
EXPECT_EQ(RefCount(wallet, Filter({k.a3}), -1, false, false, false), (size_t)2);
EXPECT_EQ(RefCount(wallet, Filter({k.a4}), -1, false, false, false), (size_t)1);
EXPECT_EQ(RefCount(wallet, Filter({k.aN}), -1, false, false, false), (size_t)2);
EXPECT_EQ(RefCount(wallet, Filter({k.empty[0]}), -1, false, false, false), (size_t)0);
EXPECT_EQ(RefCount(wallet, Filter({k.foreign}), -1, false, false, false), (size_t)0);
}
// Every filter shape x ignoreSpent x requireSpendingKey x ignoreLocked x minDepth {0, 1}.
TEST_F(WalletScanNotes, MatchesReferenceAcrossFilters)
{
CWallet wallet;
TestKeys k;
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
std::vector<NamedFilter> filters = AllFilters(k);
size_t total = 0;
for (NamedFilter& f : filters) {
for (int minDepth : {0, 1}) {
for (bool ignoreSpent : {false, true}) {
for (bool requireSpendingKey : {false, true}) {
for (bool ignoreLocked : {false, true}) {
total += ExpectMatchesReference(wallet, f, minDepth, INT_MAX,
ignoreSpent, requireSpendingKey, ignoreLocked);
}
}
}
}
}
EXPECT_GT(total, (size_t)0);
}
// The depth window: minDepth {-1, 0, 1} x maxDepth {0, 1, 3, INT_MAX} x the three flags,
// unfiltered and with a single (pre-filtered) address.
TEST_F(WalletScanNotes, MatchesReferenceAcrossDepths)
{
CWallet wallet;
TestKeys k;
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
std::vector<NamedFilter> filters;
filters.push_back(NamedFilter{"{}", Filter({})});
filters.push_back(NamedFilter{"{a1}", Filter({k.a1})});
size_t total = 0;
for (NamedFilter& f : filters) {
for (int minDepth : {-1, 0, 1}) {
for (int maxDepth : {0, 1, 3, INT_MAX}) {
for (bool ignoreSpent : {false, true}) {
for (bool requireSpendingKey : {false, true}) {
for (bool ignoreLocked : {false, true}) {
total += ExpectMatchesReference(wallet, f, minDepth, maxDepth,
ignoreSpent, requireSpendingKey, ignoreLocked);
}
}
}
}
}
}
EXPECT_GT(total, (size_t)0);
}
// The single-address string overload (z_getbalance and friends): it decodes the address
// and calls the set overload with maxDepth INT_MAX and ignoreLocked true.
TEST_F(WalletScanNotes, MatchesReferenceForAddressString)
{
CWallet wallet;
TestKeys k;
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
std::vector<std::string> addresses = {
"",
EncodePaymentAddress(k.a1),
EncodePaymentAddress(k.u1),
EncodePaymentAddress(k.x1),
EncodePaymentAddress(k.foreign),
"not-an-address",
};
size_t total = 0;
for (const std::string& address : addresses) {
for (int minDepth : {0, 1}) {
for (bool ignoreSpent : {false, true}) {
for (bool requireSpendingKey : {false, true}) {
SCOPED_TRACE(testing::Message() << "address=\"" << address << "\""
<< " minDepth=" << minDepth << " ignoreSpent=" << ignoreSpent
<< " requireSpendingKey=" << requireSpendingKey);
std::set<libzcash::PaymentAddress> filter;
if (address.length() > 0) {
filter.insert(DecodePaymentAddress(address));
}
std::vector<SaplingNoteEntry> expected;
std::vector<SaplingNoteEntry> actual;
RefGetFilteredNotes(wallet, expected, filter,
minDepth, INT_MAX, ignoreSpent, requireSpendingKey, true);
wallet.GetFilteredNotes(actual, address, minDepth, ignoreSpent, requireSpendingKey);
ExpectSameEntries(expected, actual);
total += expected.size();
}
}
}
}
EXPECT_GT(total, (size_t)0);
}
// An encrypted, locked wallet: crypted spending keys still satisfy requireSpendingKey and
// the ivk/fvk maps are not encrypted, so both implementations must still agree.
TEST_F(WalletScanNotes, MatchesReferenceWhenEncryptedAndLocked)
{
EncryptableWallet wallet;
TestKeys k;
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
CKeyingMaterial masterKey(WALLET_CRYPTO_KEY_SIZE, 0x5a);
ASSERT_TRUE(wallet.EncryptKeysForTest(masterKey));
ASSERT_TRUE(wallet.Lock());
ASSERT_TRUE(wallet.IsCrypted());
ASSERT_TRUE(wallet.IsLocked());
EXPECT_TRUE(wallet.HaveSaplingSpendingKey(k.fvk1));
std::vector<NamedFilter> filters = AllFilters(k);
size_t total = 0;
size_t spendable = 0;
for (NamedFilter& f : filters) {
for (bool ignoreSpent : {false, true}) {
for (bool requireSpendingKey : {false, true}) {
for (bool ignoreLocked : {false, true}) {
size_t n = ExpectMatchesReference(wallet, f, 0, INT_MAX,
ignoreSpent, requireSpendingKey, ignoreLocked);
total += n;
if (requireSpendingKey) {
spendable += n;
}
}
}
}
}
EXPECT_GT(total, (size_t)0);
EXPECT_GT(spendable, (size_t)0);
}
// decrypt_with_pk_d returns the same plaintext as decrypt() and the pk_d that
// ivk.address(d) recomputes, and fails exactly when decrypt() fails.
TEST_F(WalletScanNotes, DecryptWithPkDMatchesDecrypt)
{
CWallet wallet;
TestKeys k;
ASSERT_NO_FATAL_FAILURE(BuildWallet(wallet, k));
LOCK2(cs_main, wallet.cs_wallet);
size_t checked = 0;
for (const auto& p : wallet.mapWallet) {
const CWalletTx& wtx = p.second;
for (const auto& item : wtx.mapSaplingNoteData) {
SCOPED_TRACE(testing::Message() << "note " << item.first.ToString());
const OutputDescription& od = wtx.vShieldedOutput[item.first.n];
const libzcash::SaplingIncomingViewingKey& ivk = item.second.ivk;
uint256 pk_d;
auto expected = libzcash::SaplingNotePlaintext::decrypt(
od.encCiphertext, ivk, od.ephemeralKey, od.cm);
auto actual = libzcash::SaplingNotePlaintext::decrypt_with_pk_d(
od.encCiphertext, ivk, od.ephemeralKey, od.cm, pk_d);
ASSERT_TRUE(static_cast<bool>(expected));
ASSERT_TRUE(static_cast<bool>(actual));
EXPECT_TRUE(actual->d == expected->d);
EXPECT_EQ(actual->value(), expected->value());
EXPECT_EQ(actual->rcm, expected->rcm);
EXPECT_TRUE(actual->memo() == expected->memo());
auto addr = ivk.address(actual->d);
ASSERT_TRUE(static_cast<bool>(addr));
EXPECT_EQ(pk_d, addr->pk_d);
checked++;
// Wrong ivk: the AEAD fails.
const libzcash::SaplingIncomingViewingKey& wrongIvk = (ivk == k.ivk2) ? k.ivk1 : k.ivk2;
EXPECT_FALSE(DecryptBothWays(od.encCiphertext, wrongIvk, od.ephemeralKey, od.cm));
// Wrong cmu: decryption and ivk_to_pkd succeed, the commitment check fails, so
// pk_d has been computed but must not reach pk_d_out.
uint256 badCm = od.cm;
badCm.begin()[0] ^= 1;
EXPECT_FALSE(DecryptBothWays(od.encCiphertext, ivk, od.ephemeralKey, badCm));
// Tampered ciphertext.
libzcash::SaplingEncCiphertext badCt = od.encCiphertext;
badCt[0] ^= 1;
EXPECT_FALSE(DecryptBothWays(badCt, ivk, od.ephemeralKey, od.cm));
// ivk1 + r_J: whatever librustzcash decides for a non-canonical scalar, both agree.
if (ivk == k.ivk1) {
DecryptBothWays(od.encCiphertext, k.aliasIvk1, od.ephemeralKey, od.cm);
}
}
}
EXPECT_EQ(checked, k.nNotes);
}
src/init.cpp
+58 -1
View File
@@ -513,6 +513,10 @@ std::string HelpMessage(HelpMessageMode mode)
// If this is used incorrectly (-rescanheight too large), then the local wallet may attempt to spend funds which it does not have witness data about // If this is used incorrectly (-rescanheight too large), then the local wallet may attempt to spend funds which it does not have witness data about
// which will cause a "missing inputs" error when added to the mempool. Rescanning from correct height will fix this. // which will cause a "missing inputs" error when added to the mempool. Rescanning from correct height will fix this.
strUsage += HelpMessageOpt("-keepnotewitnesscache", _("Keep partial Sapling Note Witness cache. Must be used with -rescanheight to find missing cache items.")); strUsage += HelpMessageOpt("-keepnotewitnesscache", _("Keep partial Sapling Note Witness cache. Must be used with -rescanheight to find missing cache items."));
if (showDebug)
strUsage += HelpMessageOpt("-notecachewarmup", "Decrypt all wallet notes into memory at startup, so the first balance call after a restart doesn't (default: 1)");
if (showDebug)
strUsage += HelpMessageOpt("-checknotecache", "On every balance or note-listing call, decrypt each cached note it considers again and log and repair any difference from the cache (slow; default: 0)");
strUsage += HelpMessageOpt("-rescan", _("Rescan the block chain for missing wallet transactions") + " " + _("on startup")); strUsage += HelpMessageOpt("-rescan", _("Rescan the block chain for missing wallet transactions") + " " + _("on startup"));
strUsage += HelpMessageOpt("-rescanheight", _("Rescan from specified height when rescan=1 on startup")); strUsage += HelpMessageOpt("-rescanheight", _("Rescan from specified height when rescan=1 on startup"));
strUsage += HelpMessageOpt("-salvagewallet", _("Attempt to recover private keys from a corrupt wallet.dat") + " " + _("on startup")); strUsage += HelpMessageOpt("-salvagewallet", _("Attempt to recover private keys from a corrupt wallet.dat") + " " + _("on startup"));
@@ -1016,7 +1020,8 @@ extern int32_t HUSH_REWIND;
// --- Adaptive coins-cache sizing ------------------------------------------------------------- // --- Adaptive coins-cache sizing -------------------------------------------------------------
// The in-memory UTXO/coins cache (nCoinCacheUsage) is the biggest lever on IBD speed: a bigger // The in-memory UTXO/coins cache (nCoinCacheUsage) is the biggest lever on IBD speed: a bigger
// cache means far fewer chainstate flushes to disk. We size it to use most of RAM, but a scheduled // cache means fewer size-triggered chainstate flushes to disk (the chainstate is also flushed at
// least every DATABASE_FLUSH_INTERVAL / DATABASE_FLUSH_BLOCKS, see main.h). We size it to use most of RAM, but a scheduled
// background task (AdjustCoinCacheForMemoryPressure, registered in AppInit2) shrinks the target when // background task (AdjustCoinCacheForMemoryPressure, registered in AppInit2) shrinks the target when
// free system memory runs low — e.g. the user opens other apps — and grows it back when memory frees // free system memory runs low — e.g. the user opens other apps — and grows it back when memory frees
// up, always leaving a reserve free for the rest of the system. The existing per-block flush // up, always leaving a reserve free for the rest of the system. The existing per-block flush
@@ -2229,6 +2234,9 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
// ********************************************************* Step 8: load wallet // ********************************************************* Step 8: load wallet
#ifdef ENABLE_WALLET #ifdef ENABLE_WALLET
// Set when the startup rescan below is interrupted: the wallet's state then ends at the scan's
// own checkpoint, not at the tip the notifier starts from.
bool fWalletScanIncomplete = false;
if (fDisableWallet) { if (fDisableWallet) {
pwalletMain = NULL; pwalletMain = NULL;
LogPrintf("Wallet disabled!\n"); LogPrintf("Wallet disabled!\n");
@@ -2500,6 +2508,14 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
} }
} }
// Sweep and consolidation pick notes with GetFilteredNotes at minDepth > 1. Older versions
// located each note's block by re-reading its tx through the txindex, so on a -txindex=0 node
// every fully shielded tx was skipped and these never moved those notes. Now they will.
if (!fTxIndex && (pwalletMain->fSweepEnabled || pwalletMain->fSaplingConsolidationEnabled)) {
LogPrintf("%s: -txindex=0 with -zsweep/-consolidation: fully shielded notes, which versions before "
"v1.3.1 never saw on such nodes, are now included in sweeps and consolidations\n", __func__);
}
//Set Automatic Coinbase Shielding (default ON, conditional: self-guards //Set Automatic Coinbase Shielding (default ON, conditional: self-guards
//on nodes where it cannot act - no owned coinbase, external mineraddress, //on nodes where it cannot act - no owned coinbase, external mineraddress,
//or locked wallet). Closes the transparent-coinbase leak for miners. //or locked wallet). Closes the transparent-coinbase leak for miners.
@@ -2666,6 +2682,7 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
} else { } else {
LogPrintf("Rescan did not complete; leaving the best-block locator at the scan's own " LogPrintf("Rescan did not complete; leaving the best-block locator at the scan's own "
"checkpoint so the remaining range is rescanned on the next start\n"); "checkpoint so the remaining range is rescanned on the next start\n");
fWalletScanIncomplete = true;
} }
// Restore wallet transaction metadata after -zapwallettxes=1 // Restore wallet transaction metadata after -zapwallettxes=1
@@ -2693,6 +2710,15 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
} }
} }
} }
// Decrypt the wallet's notes into the in-memory plaintext cache now, while RPC is still in warmup
// and before the wallet notification thread (started below) can want cs_wallet. Otherwise the
// first balance call after a restart decrypts them all under cs_main.
fCheckNoteCache = GetBoolArg("-checknotecache", false);
if (GetBoolArg("-notecachewarmup", true) && !ShutdownRequested()) {
uiInterface.InitMessage(_("Decrypting wallet notes..."));
pwalletMain->WarmSaplingNoteCache();
}
pwalletMain->SetBroadcastTransactions(GetBoolArg("-walletbroadcast", true)); pwalletMain->SetBroadcastTransactions(GetBoolArg("-walletbroadcast", true));
} // (!fDisableWallet) } // (!fDisableWallet)
#else // ENABLE_WALLET #else // ENABLE_WALLET
@@ -2736,6 +2762,21 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
{ {
LOCK(cs_main); LOCK(cs_main);
pindexLastTip = chainActive.Tip(); pindexLastTip = chainActive.Tip();
#ifdef ENABLE_WALLET
// The wallet's state reflects the block the notifier starts from, unless the startup
// rescan was interrupted: then it ends at the scan's checkpoint, and the hourly
// SetBestChain must keep stamping that so the next start rescans the rest.
if (pwalletMain) {
const CBlockIndex* pindexWallet = pindexLastTip;
if (fWalletScanIncomplete) {
CWalletDB walletdb(pwalletMain->strWalletFile);
CBlockLocator locator;
if (walletdb.ReadBestBlock(locator))
pindexWallet = FindForkInGlobalIndex(chainActive, locator);
}
pwalletMain->SetWalletTip(pindexWallet);
}
#endif
} }
boost::function<void()> threadnotifywallets = boost::bind(&ThreadNotifyWallets, pindexLastTip); boost::function<void()> threadnotifywallets = boost::bind(&ThreadNotifyWallets, pindexLastTip);
threadGroup.create_thread( threadGroup.create_thread(
@@ -2777,8 +2818,24 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
uiInterface.InitMessage(_("Activating best chain...")); uiInterface.InitMessage(_("Activating best chain..."));
// scan for better chains in the block chain database, that are not yet connected in the active best chain // scan for better chains in the block chain database, that are not yet connected in the active best chain
CValidationState state; CValidationState state;
const CBlockIndex* pindexBeforeActivate = NULL;
{
LOCK(cs_main);
pindexBeforeActivate = chainActive.Tip();
}
if ( !ActivateBestChain(true,state)) if ( !ActivateBestChain(true,state))
strErrors << "Failed to connect best block"; strErrors << "Failed to connect best block";
else {
// Persist a startup replay now: the wallet's witness rebuild runs next and can hold
// cs_main for a long time, and a node killed before any later flush would replay it all again.
bool fTipMoved;
{
LOCK(cs_main);
fTipMoved = chainActive.Tip() != pindexBeforeActivate;
}
if (fTipMoved)
FlushStateToDisk();
}
} else { } else {
LogPrintf("HUSH_REWIND < 0\n"); LogPrintf("HUSH_REWIND < 0\n");
} }
src/main.cpp
+12 -2
View File
@@ -3738,6 +3738,7 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
static int64_t nLastWrite = 0; static int64_t nLastWrite = 0;
static int64_t nLastFlush = 0; static int64_t nLastFlush = 0;
static int64_t nLastSetChain = 0; static int64_t nLastSetChain = 0;
static int nLastFlushHeight = -1;
std::set<int> setFilesToPrune; std::set<int> setFilesToPrune;
bool fFlushForPrune = false; bool fFlushForPrune = false;
try { try {
@@ -3763,6 +3764,9 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
if (nLastSetChain == 0) { if (nLastSetChain == 0) {
nLastSetChain = nNow; nLastSetChain = nNow;
} }
if (nLastFlushHeight < 0) {
nLastFlushHeight = chainActive.Height();
}
size_t cacheSize = pcoinsTip->DynamicMemoryUsage(); size_t cacheSize = pcoinsTip->DynamicMemoryUsage();
// The cache is large and close to the limit, but we have time now (not in the middle of a block processing). // The cache is large and close to the limit, but we have time now (not in the middle of a block processing).
bool fCacheLarge = mode == FLUSH_STATE_PERIODIC && cacheSize * (10.0/9) > nCoinCacheUsage; bool fCacheLarge = mode == FLUSH_STATE_PERIODIC && cacheSize * (10.0/9) > nCoinCacheUsage;
@@ -3772,8 +3776,10 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
bool fPeriodicWrite = mode == FLUSH_STATE_PERIODIC && nNow > nLastWrite + (int64_t)DATABASE_WRITE_INTERVAL * 1000000; bool fPeriodicWrite = mode == FLUSH_STATE_PERIODIC && nNow > nLastWrite + (int64_t)DATABASE_WRITE_INTERVAL * 1000000;
// It's been very long since we flushed the cache. Do this infrequently, to optimize cache usage. // It's been very long since we flushed the cache. Do this infrequently, to optimize cache usage.
bool fPeriodicFlush = mode == FLUSH_STATE_PERIODIC && nNow > nLastFlush + (int64_t)DATABASE_FLUSH_INTERVAL * 1000000; bool fPeriodicFlush = mode == FLUSH_STATE_PERIODIC && nNow > nLastFlush + (int64_t)DATABASE_FLUSH_INTERVAL * 1000000;
// Many blocks connected since the last flush, e.g. a long replay inside one ActivateBestChain call.
bool fBlockCountFlush = (mode == FLUSH_STATE_IF_NEEDED || mode == FLUSH_STATE_PERIODIC) && chainActive.Height() - nLastFlushHeight >= DATABASE_FLUSH_BLOCKS;
// Combine all conditions that result in a full cache flush. // Combine all conditions that result in a full cache flush.
bool fDoFullFlush = (mode == FLUSH_STATE_ALWAYS) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune; bool fDoFullFlush = (mode == FLUSH_STATE_ALWAYS) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune || fBlockCountFlush;
// Write blocks and block index to disk. // Write blocks and block index to disk.
if (fDoFullFlush || fPeriodicWrite) { if (fDoFullFlush || fPeriodicWrite) {
// Depend on nMinDiskSpace to ensure we can write block index // Depend on nMinDiskSpace to ensure we can write block index
@@ -3827,6 +3833,7 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
if (!pcoinsTip->Flush()) if (!pcoinsTip->Flush())
return AbortNode(state, "Failed to write to coin database"); return AbortNode(state, "Failed to write to coin database");
nLastFlush = nNow; nLastFlush = nNow;
nLastFlushHeight = chainActive.Height();
} }
if ((mode == FLUSH_STATE_ALWAYS || mode == FLUSH_STATE_PERIODIC) && nNow > nLastSetChain + (int64_t)DATABASE_WRITE_INTERVAL * 1000000) { if ((mode == FLUSH_STATE_ALWAYS || mode == FLUSH_STATE_PERIODIC) && nNow > nLastSetChain + (int64_t)DATABASE_WRITE_INTERVAL * 1000000) {
// Update best block in wallet (so we can detect restored wallets). // Update best block in wallet (so we can detect restored wallets).
@@ -5915,7 +5922,10 @@ FILE* OpenDiskFile(const CDiskBlockPos &pos, const char *prefix, bool fReadOnly)
return NULL; return NULL;
boost::filesystem::path path = GetBlockPosFilename(pos, prefix); boost::filesystem::path path = GetBlockPosFilename(pos, prefix);
boost::filesystem::create_directories(path.parent_path()); boost::filesystem::create_directories(path.parent_path());
FILE* file = fopen(path.string().c_str(), "rb+"); // Read-only opens get a read-only FILE (every fReadOnly caller only reads). With "rb+", a write through a
// reused fd number landed here: when the wolfSSL CloseSocket macro (see net.h) closed the wallet's BDB fd
// behind its back, BDB's next page write went to whatever reopened that number -- a block file opened here.
FILE* file = fopen(path.string().c_str(), fReadOnly ? "rb" : "rb+");
if (!file && !fReadOnly) if (!file && !fReadOnly)
file = fopen(path.string().c_str(), "wb+"); file = fopen(path.string().c_str(), "wb+");
if (!file) { if (!file) {
src/main.h
+9 -2
View File
@@ -143,8 +143,15 @@ static const unsigned int MAX_LOCATOR_SZ = 101;
static unsigned int BLOCK_DOWNLOAD_WINDOW = 1024; static unsigned int BLOCK_DOWNLOAD_WINDOW = 1024;
/** Time to wait (in seconds) between writing blocks/block index to disk. */ /** Time to wait (in seconds) between writing blocks/block index to disk. */
static const unsigned int DATABASE_WRITE_INTERVAL = 60 * 60; static const unsigned int DATABASE_WRITE_INTERVAL = 60 * 60;
/** Time to wait (in seconds) between flushing chainstate to disk. */ /** Time to wait (in seconds) between flushing chainstate to disk. Upstream uses 24h to keep a large
static const unsigned int DATABASE_FLUSH_INTERVAL = 24 * 60 * 60; * coins cache warm, but DragonX's cache stays a few MiB (private chain, few transparent outputs), so
* the size triggers never fire and this timer is all that persists the chainstate while a node runs:
* a node that never reached 24h uptime or a clean shutdown replayed weeks of blocks on every start. */
static const unsigned int DATABASE_FLUSH_INTERVAL = 60 * 60;
/** Also flush chainstate once this many blocks were connected since the last flush. A long
* ActivateBestChain (startup replay, catching up after downtime) only flushes IF_NEEDED until it
* returns, so without this a node killed part-way through loses the whole replay. */
static const int DATABASE_FLUSH_BLOCKS = 5000;
/** Maximum length of reject messages. */ /** Maximum length of reject messages. */
static const unsigned int MAX_REJECT_MESSAGE_LENGTH = 111; static const unsigned int MAX_REJECT_MESSAGE_LENGTH = 111;
static const int64_t DEFAULT_MAX_TIP_AGE = 24 * 60 * 60; static const int64_t DEFAULT_MAX_TIP_AGE = 24 * 60 * 60;
src/net.cpp
+7 -10
View File
@@ -45,6 +45,13 @@
#include <wolfssl/options.h> #include <wolfssl/options.h>
#include <wolfssl/ssl.h> #include <wolfssl/ssl.h>
#include <hush/tlsmanager.cpp> #include <hush/tlsmanager.cpp>
// CloseSocket must be util/sock.cpp's CloseSocket(SOCKET&), which invalidates the handle it closes. If
// wolfSSL's `#define CloseSocket(s) close(s)` survived into this file (see net.h), each close here would leave
// its stale fd number behind to be closed again after another thread reused it.
#ifdef CloseSocket
#error "CloseSocket is a macro here (wolfssl/wolfio.h): it would bypass CloseSocket(SOCKET&) in util/sock.cpp"
#endif
using namespace hush; using namespace hush;
extern int32_t HUSH_TESTNODE; extern int32_t HUSH_TESTNODE;
extern uint8_t ASSETCHAINS_CLEARNET; extern uint8_t ASSETCHAINS_CLEARNET;
@@ -2951,16 +2958,6 @@ void CNode::PushAddrMessage(CSerializedNetMsg&& msg)
} }
} }
void GetNodeCounts(int& nAll, int& nTls)
{
LOCK(cs_vNodes);
nAll = (int)vNodes.size();
nTls = 0;
for (const CNode* n : vNodes)
if (n->ssl != NULL)
nTls++;
}
size_t GetNodeCount(NumConnections flags) size_t GetNodeCount(NumConnections flags)
{ {
LOCK(cs_vNodes); LOCK(cs_vNodes);
src/net.h
+8 -1
View File
@@ -57,6 +57,14 @@
#define byte unsigned char #define byte unsigned char
#include <wolfssl/ssl.h> #include <wolfssl/ssl.h>
#undef byte #undef byte
// wolfssl/wolfio.h (pulled in by ssl.h) does `#ifndef CloseSocket / #define CloseSocket(s) close(s)`. Left
// defined, it turns every CloseSocket(hSocket) after this point -- all 15 calls in net.cpp, including the ones in
// CloseSocketDisconnect and ~CNode -- into a bare close() that never sets hSocket = INVALID_SOCKET, so util/sock.cpp's
// CloseSocket(SOCKET&) is never called. Every P2P disconnect then closed its fd number again later (a repeated
// CloseSocketDisconnect, then ~CNode), by which time another thread often owned that number: wallet.dat or a
// database/log.* fd (BDB then wrote wallet pages into whatever got the number next, e.g. a blocks/*.dat file
// opened "rb+"), a block file, or an HTTP RPC socket. wolfio.h is include-guarded, so it can't come back.
#undef CloseSocket
class CAddrMan; class CAddrMan;
class CBlockIndex; class CBlockIndex;
@@ -165,7 +173,6 @@ enum NumConnections {
}; };
size_t GetNodeCount(NumConnections num); size_t GetNodeCount(NumConnections num);
void GetNodeCounts(int& nAll, int& nTls);
bool GetNetworkActive(); bool GetNetworkActive();
void SetNetworkActive(bool active); void SetNetworkActive(bool active);
src/rpc/misc.cpp
+2 -6
View File
@@ -132,10 +132,6 @@ UniValue getinfo(const UniValue& params, bool fHelp, const CPubKey& mypk)
+ HelpExampleRpc("getinfo", "") + HelpExampleRpc("getinfo", "")
); );
// C268: snapshot peer counts under cs_vNodes (sequentially, before the cs_main/cs_wallet lock)
int nAllConns = 0, nTlsConns = 0;
GetNodeCounts(nAllConns, nTlsConns);
#ifdef ENABLE_WALLET #ifdef ENABLE_WALLET
LOCK2(cs_main, pwalletMain ? &pwalletMain->cs_wallet : NULL); LOCK2(cs_main, pwalletMain ? &pwalletMain->cs_wallet : NULL);
#else #else
@@ -179,8 +175,8 @@ UniValue getinfo(const UniValue& params, bool fHelp, const CPubKey& mypk)
} }
//obj.push_back(Pair("build_date", BUILD_DATE)); //obj.push_back(Pair("build_date", BUILD_DATE));
obj.push_back(Pair("timeoffset", 0)); obj.push_back(Pair("timeoffset", 0));
obj.push_back(Pair("connections", nAllConns)); obj.push_back(Pair("connections", (int)vNodes.size()));
obj.push_back(Pair("tls_connections", nTlsConns)); obj.push_back(Pair("tls_connections", (int)std::count_if(vNodes.begin(), vNodes.end(), [](CNode* n) {return n->ssl != NULL;})));
obj.push_back(Pair("proxy", (proxy.IsValid() ? proxy.proxy.ToStringIPPort() : string()))); obj.push_back(Pair("proxy", (proxy.IsValid() ? proxy.proxy.ToStringIPPort() : string())));
obj.push_back(Pair("testnet", Params().TestnetToBeDeprecatedFieldRPC())); obj.push_back(Pair("testnet", Params().TestnetToBeDeprecatedFieldRPC()));
obj.push_back(Pair("relayfee", ValueFromAmount(::minRelayTxFee.GetFeePerK()))); obj.push_back(Pair("relayfee", ValueFromAmount(::minRelayTxFee.GetFeePerK())));
src/rpc/net.cpp
+2 -5
View File
@@ -533,9 +533,6 @@ UniValue getnetworkinfo(const UniValue& params, bool fHelp, const CPubKey& mypk)
+ HelpExampleRpc("getnetworkinfo", "") + HelpExampleRpc("getnetworkinfo", "")
); );
int nAllConns = 0, nTlsConns = 0;
GetNodeCounts(nAllConns, nTlsConns);
LOCK(cs_main); LOCK(cs_main);
UniValue obj(UniValue::VOBJ); UniValue obj(UniValue::VOBJ);
@@ -544,8 +541,8 @@ UniValue getnetworkinfo(const UniValue& params, bool fHelp, const CPubKey& mypk)
obj.push_back(Pair("protocolversion",PROTOCOL_VERSION)); obj.push_back(Pair("protocolversion",PROTOCOL_VERSION));
obj.push_back(Pair("localservices", strprintf("%016x", nLocalServices))); obj.push_back(Pair("localservices", strprintf("%016x", nLocalServices)));
obj.push_back(Pair("timeoffset", 0)); obj.push_back(Pair("timeoffset", 0));
obj.push_back(Pair("connections", nAllConns)); obj.push_back(Pair("connections", (int)vNodes.size()));
obj.push_back(Pair("tls_connections", nTlsConns)); obj.push_back(Pair("tls_connections", (int)std::count_if(vNodes.begin(), vNodes.end(), [](CNode* n) {return n->ssl != NULL;})));
obj.push_back(Pair("networks", GetNetworksInfo())); obj.push_back(Pair("networks", GetNetworksInfo()));
obj.push_back(Pair("relayfee", ValueFromAmount(::minRelayTxFee.GetFeePerK()))); obj.push_back(Pair("relayfee", ValueFromAmount(::minRelayTxFee.GetFeePerK())));
UniValue localAddresses(UniValue::VARR); UniValue localAddresses(UniValue::VARR);
src/rpc/server.cpp
+26
View File
@@ -26,6 +26,7 @@
#include "util.h" #include "util.h"
#include "util/strencodings.h" #include "util/strencodings.h"
#include "asyncrpcqueue.h" #include "asyncrpcqueue.h"
#include <chrono>
#include <memory> #include <memory>
#include <univalue.h> #include <univalue.h>
#include <boost/bind.hpp> #include <boost/bind.hpp>
@@ -640,6 +641,30 @@ std::string JSONRPCExecBatch(const UniValue& vReq)
return ret.write() + "\n"; return ret.write() + "\n";
} }
namespace {
// Logs an RPC whose wall time reached 500ms, or every RPC under -debug=bench. That time includes
// waiting for cs_main / cs_wallet behind block connection or another RPC, which the callee's own
// timers can't see because they start once the locks are held. A long-poll getblocktemplate waits
// for a new block with cs_main released, so it is only logged under -debug=bench. The method is
// logged as method=<name>: a bare " <name> " token would match the GUI's " rescan " log heuristic.
class RPCCallTimer {
const std::string& strMethod;
const bool fLongPoll;
const std::chrono::steady_clock::time_point start;
public:
RPCCallTimer(const std::string& method, const UniValue& params)
: strMethod(method),
fLongPoll(method == "getblocktemplate" && params.size() > 0 && params[0].isObject() &&
!find_value(params[0], "longpollid").isNull()),
start(std::chrono::steady_clock::now()) {}
~RPCCallTimer() {
const int64_t nMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - start).count();
if ((nMs >= 500 && !fLongPoll) || LogAcceptCategory("bench"))
LogPrintf("RPC method=%s took %dms\n", strMethod, nMs);
}
};
}
UniValue CRPCTable::execute(const std::string &strMethod, const UniValue &params) const UniValue CRPCTable::execute(const std::string &strMethod, const UniValue &params) const
{ {
// Find method // Find method
@@ -674,6 +699,7 @@ UniValue CRPCTable::execute(const std::string &strMethod, const UniValue &params
try try
{ {
// Execute // Execute
RPCCallTimer timer(strMethod, params);
return pcmd->actor(params, false, CPubKey()); return pcmd->actor(params, false, CPubKey());
} }
catch (const std::exception& e) catch (const std::exception& e)
src/validationinterface.cpp
+53 -18
View File
@@ -86,6 +86,30 @@ struct CachedBlockData {
pindex(pindex), oldTrees(oldTrees), txConflicted(txConflicted) {} pindex(pindex), oldTrees(oldTrees), txConflicted(txConflicted) {}
}; };
bool NotifyWalletsOrShutdown(const std::function<void()>& notify)
{
// The wallet can throw here on a database failure, e.g. CWalletDB::ListAccountCreditDebit's "error scanning
// DB" after BDB lost its file. Escaping this thread, that std::terminate()d the node (TraceThread logs and
// rethrows) with nothing flushed. Stop cleanly instead: going on would leave the wallet behind the chain,
// missing the block it failed on.
try {
notify();
return true;
} catch (const boost::thread_interrupted&) {
throw;
} catch (const std::exception& e) {
PrintExceptionContinue(&e, "ThreadNotifyWallets()");
} catch (...) {
PrintExceptionContinue(NULL, "ThreadNotifyWallets()");
}
LogPrintf("*** %s\n", "Wallet failed to process a block notification; shutting down");
StartShutdown();
uiInterface.ThreadSafeMessageBox(
_("Error: A fatal internal error occurred, see debug.log for details"),
"", CClientUIInterface::MSG_ERROR);
return false;
}
void ThreadNotifyWallets(CBlockIndex *pindexLastTip) void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
{ {
// If pindexLastTip == nullptr, the wallet is at genesis. // If pindexLastTip == nullptr, the wallet is at genesis.
@@ -203,15 +227,21 @@ void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
_("Error: A fatal internal error occurred, see debug.log for details"), _("Error: A fatal internal error occurred, see debug.log for details"),
"", CClientUIInterface::MSG_ERROR); "", CClientUIInterface::MSG_ERROR);
StartShutdown(); StartShutdown();
// Don't hand the wallet an empty block as if it had been processed.
fDisconnectIncomplete = true;
break;
} }
// Let wallets know transactions went from 1-confirmed to if (!NotifyWalletsOrShutdown([&] {
// 0-confirmed or conflicted: // Let wallets know transactions went from 1-confirmed to
for (const CTransaction &tx : block.vtx) { // 0-confirmed or conflicted:
SyncWithWallets(tx, NULL); for (const CTransaction &tx : block.vtx) {
} SyncWithWallets(tx, NULL);
// Update cached incremental witnesses }
GetMainSignals().ChainTip(pindexLastTip, &block, boost::none); // Update cached incremental witnesses
GetMainSignals().ChainTip(pindexLastTip, &block, boost::none);
}))
return;
// On to the next block! // On to the next block!
pindexLastTip = pindexLastTip->pprev; pindexLastTip = pindexLastTip->pprev;
@@ -239,19 +269,24 @@ void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
_("Error: A fatal internal error occurred, see debug.log for details"), _("Error: A fatal internal error occurred, see debug.log for details"),
"", CClientUIInterface::MSG_ERROR); "", CClientUIInterface::MSG_ERROR);
StartShutdown(); StartShutdown();
// Don't hand the wallet an empty block as if it had been processed.
break;
} }
// Tell wallet about transactions that went from mempool if (!NotifyWalletsOrShutdown([&] {
// to conflicted: // Tell wallet about transactions that went from mempool
for (const CTransaction &tx : blockData.txConflicted) { // to conflicted:
SyncWithWallets(tx, NULL); for (const CTransaction &tx : blockData.txConflicted) {
} SyncWithWallets(tx, NULL);
// ... and about transactions that got confirmed: }
for (const CTransaction &tx : block.vtx) { // ... and about transactions that got confirmed:
SyncWithWallets(tx, &block); for (const CTransaction &tx : block.vtx) {
} SyncWithWallets(tx, &block);
// Update cached incremental witnesses }
GetMainSignals().ChainTip(blockData.pindex, &block, blockData.oldTrees); // Update cached incremental witnesses
GetMainSignals().ChainTip(blockData.pindex, &block, blockData.oldTrees);
}))
return;
// This block is done! // This block is done!
pindexLastTip = blockData.pindex; pindexLastTip = blockData.pindex;
src/validationinterface.h
+5
View File
@@ -8,6 +8,7 @@
#define HUSH_VALIDATIONINTERFACE_H #define HUSH_VALIDATIONINTERFACE_H
#include <boost/signals2/signal.hpp> #include <boost/signals2/signal.hpp>
#include <functional>
#include "zcash/IncrementalMerkleTree.hpp" #include "zcash/IncrementalMerkleTree.hpp"
class CBlock; class CBlock;
@@ -76,4 +77,8 @@ CMainSignals& GetMainSignals();
void ThreadNotifyWallets(CBlockIndex *pindexLastTip); void ThreadNotifyWallets(CBlockIndex *pindexLastTip);
/** Runs one block's wallet notifications for ThreadNotifyWallets. Returns false, after logging the error and
* requesting a shutdown, if they threw (e.g. a wallet database failure); thread interruption propagates. */
bool NotifyWalletsOrShutdown(const std::function<void()>& notify);
#endif // HUSH_VALIDATIONINTERFACE_H #endif // HUSH_VALIDATIONINTERFACE_H
src/version.h
+7 -1
View File
@@ -22,7 +22,13 @@
// network protocol versioning // network protocol versioning
// DragonX 1.0.0 - bumped to separate from old HUSH/DragonX nodes with RandomX bug // DragonX 1.0.0 - bumped to separate from old HUSH/DragonX nodes with RandomX bug
static const int PROTOCOL_VERSION = 2000000; // DragonX 1.3.0 - bumped for the consensus changes in this release (BLOCK_VALID_CONTEXT
// no longer overwrites the validity level, the dead CBOPRET coinbase
// price check is gone, and pow.cpp lost 157 lines). MIN_PEER_PROTO_VERSION
// is deliberately NOT raised: it currently equals this value, and raising
// it would disconnect every node still on v1.0.3 -- which is all users,
// since no release has ever been installable (unsigned archives).
static const int PROTOCOL_VERSION = 2000001;
//! initial proto version, to be increased after version/verack negotiation //! initial proto version, to be increased after version/verack negotiation
static const int INIT_PROTO_VERSION = 209; static const int INIT_PROTO_VERSION = 209;
//! In this version, 'getheaders' was introduced. //! In this version, 'getheaders' was introduced.
src/wallet/crypter.cpp
+1 -1
View File
@@ -591,7 +591,7 @@ bool CCryptoKeyStore::GetSaplingSpendingKey(const libzcash::SaplingFullViewingKe
if (!IsCrypted()) if (!IsCrypted())
return CBasicKeyStore::GetSaplingSpendingKey(fvk, skOut); return CBasicKeyStore::GetSaplingSpendingKey(fvk, skOut);
for (auto entry : mapCryptedSaplingSpendingKeys) { for (const auto& entry : mapCryptedSaplingSpendingKeys) {
if (entry.first.fvk == fvk) { if (entry.first.fvk == fvk) {
const std::vector<unsigned char> &vchCryptedSecret = entry.second; const std::vector<unsigned char> &vchCryptedSecret = entry.second;
return DecryptSaplingSpendingKey(vMasterKey, vchCryptedSecret, entry.first, skOut); return DecryptSaplingSpendingKey(vMasterKey, vchCryptedSecret, entry.first, skOut);
src/wallet/crypter.h
+1 -1
View File
@@ -265,7 +265,7 @@ public:
LOCK(cs_SpendingKeyStore); LOCK(cs_SpendingKeyStore);
if (!IsCrypted()) if (!IsCrypted())
return CBasicKeyStore::HaveSaplingSpendingKey(fvk); return CBasicKeyStore::HaveSaplingSpendingKey(fvk);
for (auto entry : mapCryptedSaplingSpendingKeys) { for (const auto& entry : mapCryptedSaplingSpendingKeys) {
if (entry.first.fvk == fvk) { if (entry.first.fvk == fvk) {
return true; return true;
} }
src/wallet/db.cpp
+32 -9
View File
@@ -61,6 +61,7 @@ void CDBEnv::Reset()
dbenv = new DbEnv(DB_CXX_NO_EXCEPTIONS); dbenv = new DbEnv(DB_CXX_NO_EXCEPTIONS);
fDbEnvInit = false; fDbEnvInit = false;
fMockDb = false; fMockDb = false;
fKeepLogs = false;
} }
CDBEnv::CDBEnv() : dbenv(NULL) CDBEnv::CDBEnv() : dbenv(NULL)
@@ -276,10 +277,12 @@ bool CDBEnv::Salvage(const std::string& strFile, bool fAggressive, std::vector<C
void CDBEnv::CheckpointLSN(const std::string& strFile) void CDBEnv::CheckpointLSN(const std::string& strFile)
{ {
dbenv->txn_checkpoint(0, 0, 0); if (dbenv->txn_checkpoint(0, 0, 0) != 0)
fKeepLogs = true;
if (fMockDb) if (fMockDb)
return; return;
dbenv->lsn_reset(strFile.c_str(), 0); if (dbenv->lsn_reset(strFile.c_str(), 0) != 0)
fKeepLogs = true;
} }
@@ -498,10 +501,16 @@ void CDBEnv::Flush(bool fShutdown)
// Move log data to the dat file // Move log data to the dat file
CloseDb(strFile); CloseDb(strFile);
LogPrint("db", "CDBEnv::Flush: %s checkpoint\n", strFile); LogPrint("db", "CDBEnv::Flush: %s checkpoint\n", strFile);
dbenv->txn_checkpoint(0, 0, 0); int ret = dbenv->txn_checkpoint(0, 0, 0);
if (ret != 0) {
LogPrintf("CDBEnv::Flush: %s checkpoint failed: %s\n", strFile, DbEnv::strerror(ret));
fKeepLogs = true;
}
LogPrint("db", "CDBEnv::Flush: %s detach\n", strFile); LogPrint("db", "CDBEnv::Flush: %s detach\n", strFile);
if (!fMockDb) if (!fMockDb && (ret = dbenv->lsn_reset(strFile.c_str(), 0)) != 0) {
dbenv->lsn_reset(strFile.c_str(), 0); LogPrintf("CDBEnv::Flush: %s lsn_reset failed: %s\n", strFile, DbEnv::strerror(ret));
fKeepLogs = true;
}
LogPrint("db", "CDBEnv::Flush: %s closed\n", strFile); LogPrint("db", "CDBEnv::Flush: %s closed\n", strFile);
mapFileUseCount.erase(mi++); mapFileUseCount.erase(mi++);
} else } else
@@ -511,10 +520,24 @@ void CDBEnv::Flush(bool fShutdown)
if (fShutdown) { if (fShutdown) {
char** listp; char** listp;
if (mapFileUseCount.empty()) { if (mapFileUseCount.empty()) {
dbenv->log_archive(&listp, DB_ARCH_REMOVE); // After a failed flush, or once the environment has panicked (BDB0060, e.g. after a page
Close(); // write failed), wallet.dat can be missing committed changes that only the logs hold.
if (!fMockDb) // Removing them would turn a wallet the next start recovers (DB_RECOVER) into one that
boost::filesystem::remove_all(boost::filesystem::path(strPath) / "database"); // silently lost them. If the logs are unusable, the next start moves database/ aside.
// (get_flags itself fails with DB_RUNRECOVERY once the environment has panicked.)
u_int32_t envFlags = 0;
if (dbenv->get_flags(&envFlags) != 0 || (envFlags & DB_PANIC_ENVIRONMENT))
fKeepLogs = true;
if (fKeepLogs && !fMockDb) {
LogPrintf("CDBEnv::Flush: the wallet database was not flushed cleanly; keeping %s for recovery at the next start\n",
(boost::filesystem::path(strPath) / "database").string());
Close();
} else {
dbenv->log_archive(&listp, DB_ARCH_REMOVE);
Close();
if (!fMockDb)
boost::filesystem::remove_all(boost::filesystem::path(strPath) / "database");
}
} }
} }
} }
src/wallet/db.h
+4
View File
@@ -28,6 +28,7 @@
#include "sync.h" #include "sync.h"
#include "version.h" #include "version.h"
#include <atomic>
#include <map> #include <map>
#include <string> #include <string>
#include <vector> #include <vector>
@@ -53,6 +54,9 @@ class CDBEnv
private: private:
bool fDbEnvInit; bool fDbEnvInit;
bool fMockDb; bool fMockDb;
// A checkpoint or lsn_reset failed: the logs in database/ may hold the only copy of committed
// wallet changes, so the shutdown flush must leave them for DB_RECOVER at the next start.
std::atomic<bool> fKeepLogs;
// Don't change into boost::filesystem::path, as that can result in // Don't change into boost::filesystem::path, as that can result in
// shutdown problems/crashes caused by a static initialized internal pointer. // shutdown problems/crashes caused by a static initialized internal pointer.
std::string strPath; std::string strPath;
src/wallet/rpchushwallet.cpp
+155 -139
View File
@@ -11,70 +11,112 @@
#include "utilmoneystr.h" #include "utilmoneystr.h"
#include <utf8.h> #include <utf8.h>
#include <map>
#include <set>
#include <utility>
#include <vector>
using namespace std; using namespace std;
using namespace libzcash; using namespace libzcash;
bool EnsureWalletIsAvailable(bool avoidException); bool EnsureWalletIsAvailable(bool avoidException);
const std::vector<ZsTxJSONContext::Key>& ZsTxJSONContext::keys() SaplingHistoryKeys GetSaplingHistoryKeys()
{ {
if (!keysBuilt) { AssertLockHeld(pwalletMain->cs_wallet);
// The calls, order and filter of the per-output key loops this replaces. SaplingHistoryKeys keys;
std::set<libzcash::SaplingPaymentAddress> addresses; std::set<libzcash::SaplingPaymentAddress> addresses;
pwalletMain->GetSaplingPaymentAddresses(addresses); pwalletMain->GetSaplingPaymentAddresses(addresses);
for (std::set<libzcash::SaplingPaymentAddress>::const_iterator it = addresses.begin(); it != addresses.end(); ++it) { for (const auto& addr : addresses) {
libzcash::SaplingExtendedSpendingKey extsk; libzcash::SaplingExtendedSpendingKey extsk;
if (!pwalletMain->GetSaplingExtendedSpendingKey(*it, extsk)) if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
continue; // viewing-key-only, or encrypted and locked: skipped, as before const libzcash::SaplingFullViewingKey fvk = extsk.expsk.full_viewing_key();
libzcash::SaplingFullViewingKey fvk = extsk.expsk.full_viewing_key(); SaplingHistoryKeys::SpendableAddress a;
Key key; a.addr = addr;
key.addr = *it; a.ivk = fvk.in_viewing_key();
key.ivk = fvk.in_viewing_key(); a.ovk = fvk.ovk;
key.ovk = fvk.ovk; // Left zero when the wallet doesn't know addr, exactly as GetNullifiersForAddresses does.
vKeys.push_back(key); pwalletMain->GetSaplingIncomingViewingKey(addr, a.walletIvk);
} keys.spendable.push_back(a);
keysBuilt = true;
} }
return vKeys; }
return keys;
} }
bool ZsTxJSONContext::hasNullifier(const libzcash::SaplingPaymentAddress& addr, const uint256& nullifier) bool SaplingHistoryKeys::IsNullifierOfAddress(const libzcash::SaplingIncomingViewingKey& addrIvk, const uint256& nullifier) const
{ {
if (!nullifiersBuilt) { AssertLockHeld(pwalletMain->cs_wallet);
// The population GetNullifiersForAddresses() scans: every mapWallet note that has a nullifier. if (!noteNullifiers) {
for (std::map<uint256, CWalletTx>::const_iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) { noteNullifiers = std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256>>();
for (mapSaplingNoteData_t::const_iterator nit = it->second.mapSaplingNoteData.begin(); nit != it->second.mapSaplingNoteData.end(); ++nit) { for (const auto& p : pwalletMain->mapWallet) {
if (nit->second.nullifier) for (const auto& item : p.second.mapSaplingNoteData) {
nfByIvk.insert(std::make_pair(nit->second.ivk, nit->second.nullifier.get())); if (item.second.nullifier)
} noteNullifiers->insert(std::make_pair(item.second.ivk, *item.second.nullifier));
} }
nullifiersBuilt = true;
} }
// GetNullifiersForAddresses() ignores the result of this lookup, so an address the wallet does not }
// know is matched against the default (null) ivk. Keep that. return noteNullifiers->count(std::make_pair(addrIvk, nullifier)) > 0;
libzcash::SaplingIncomingViewingKey ivk;
pwalletMain->GetSaplingIncomingViewingKey(addr, ivk);
return nfByIvk.count(std::make_pair(ivk, nullifier)) > 0;
} }
bool ZsTxJSONContext::seedOvk(uint256& ovkOut) namespace {
{
if (!seedOvkTried) { // One output's decryptions under incoming viewing keys, each key tried at most once. Two distinct ivks can't
HDSeed seed; // both decrypt an output: the cm check recomputes pk_d = [ivk]g_d, and ivks are canonical scalars below the
if (pwalletMain->GetHDSeedForDerivation(seed)) { // order of g_d (ka_agree and ivk_to_pkd reject the rest), so a second one would need a NoteCommit collision.
seedOvkValue = ovkForShieldingFromTaddr(seed); // So once it is known which ivk decrypts the output (the wallet's note data records the one FindMySaplingNotes
seedOvkOk = true; // found, with its plaintext normally cached), no other ivk is tried.
} class OutputDecryptions {
seedOvkTried = true; const OutputDescription& output;
const SaplingNoteData* nd;
std::map<libzcash::SaplingIncomingViewingKey, boost::optional<SaplingNoteCache>> tried;
boost::optional<SaplingNoteCache> Decrypt(const libzcash::SaplingIncomingViewingKey& ivk) const {
auto both = libzcash::SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
if (!both)
return boost::none;
return SaplingNoteCache{ both->second, both->first.memo() };
}
// The plaintext under the ivk the wallet recorded for this output, decrypting it if it isn't cached.
const boost::optional<SaplingNoteCache>& Recorded() {
auto it = tried.find(nd->ivk);
if (it == tried.end())
it = tried.insert(std::make_pair(nd->ivk, nd->cachedPlaintext ? nd->cachedPlaintext : Decrypt(nd->ivk))).first;
return it->second;
}
public:
OutputDecryptions(const CWalletTx& wtx, uint32_t n) : output(wtx.vShieldedOutput[n]), nd(nullptr) {
auto it = wtx.mapSaplingNoteData.find(SaplingOutPoint(wtx.GetHash(), n));
if (it != wtx.mapSaplingNoteData.end())
nd = &it->second;
}
const SaplingNoteCache* Get(const libzcash::SaplingIncomingViewingKey& ivk) {
auto it = tried.find(ivk);
if (it == tried.end()) {
boost::optional<SaplingNoteCache> result;
if (nd && nd->ivk == ivk) {
result = Recorded();
} else if (!(nd && Recorded())) {
// Not a wallet note, or one whose recorded ivk doesn't decrypt it: try this key.
result = Decrypt(ivk);
}
it = tried.insert(std::make_pair(ivk, result)).first;
} }
if (seedOvkOk) return it->second ? &*it->second : nullptr;
ovkOut = seedOvkValue; }
return seedOvkOk; };
// What an outCiphertext tells the sender holding ovk: the note's plaintext and the address it was sent to.
boost::optional<std::pair<libzcash::SaplingNotePlaintext, libzcash::SaplingPaymentAddress>> DecryptSent(
const OutputDescription& outputDesc, const uint256& ovk)
{
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
outputDesc.outCiphertext,ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
if (!opt)
return boost::none;
auto opt_unwrapped = opt.get();
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
if (!pt)
return boost::none;
auto pt_unwrapped = pt.get();
return std::make_pair(pt_unwrapped, libzcash::SaplingPaymentAddress(pt_unwrapped.d, opt_unwrapped.pk_d));
}
} }
void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) { void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) {
@@ -148,22 +190,29 @@ void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpen
const SpendDescription& spendDesc = wtx.vShieldedSpend[i]; const SpendDescription& spendDesc = wtx.vShieldedSpend[i];
UniValue obj(UniValue::VOBJ); UniValue obj(UniValue::VOBJ);
std::map<uint256, SaplingOutPoint>::const_iterator nfIt = pwalletMain->mapSaplingNullifiersToNotes.find(spendDesc.nullifier); // Lookups by find(), not operator[]: that inserted an entry for every other wallet's nullifier it met
// (and could insert an empty note into a wallet tx).
// Same test as IsSaplingNullifierFromMe(), but find() so a read-only RPC never inserts into wallet maps. if (pwalletMain->IsSaplingNullifierFromMe(spendDesc.nullifier)) {
if (nfIt != pwalletMain->mapSaplingNullifiersToNotes.end() && pwalletMain->mapWallet.count(nfIt->second.hash)) { const SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes.at(spendDesc.nullifier);
const SaplingOutPoint op = nfIt->second;
const CWalletTx* parent = pwalletMain->GetWalletTx(op.hash); const CWalletTx* parent = pwalletMain->GetWalletTx(op.hash);
mapSaplingNoteData_t::const_iterator ndIt = parent->mapSaplingNoteData.find(op); auto ndIt = parent->mapSaplingNoteData.find(op);
if (ndIt == parent->mapSaplingNoteData.end()) continue; // stale entry: old code inserted an empty note that never decrypts // The spent note's plaintext, from the wallet's cache when it has it.
const OutputDescription& output = parent->vShieldedOutput[op.n]; boost::optional<SaplingNoteCache> pt;
const SaplingNoteData& nd = ndIt->second; if (ndIt != parent->mapSaplingNoteData.end() && op.n < parent->vShieldedOutput.size()) {
auto pt = libzcash::SaplingNotePlaintext::decrypt(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm); const SaplingNoteData& nd = ndIt->second;
if (nd.cachedPlaintext) {
pt = nd.cachedPlaintext;
} else {
const OutputDescription& output = parent->vShieldedOutput[op.n];
auto both = libzcash::SaplingNotePlaintext::decrypt_with_note(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
if (both)
pt = SaplingNoteCache{ both->second, both->first.memo() };
}
}
if (pt) { if (pt) {
auto note = pt.get(); const libzcash::SaplingNote& note = pt->note;
auto pa = nd.ivk.address(note.d); const libzcash::SaplingPaymentAddress address(note.d, note.pk_d);
auto address = pa.get();
shieldedSpendsSpentCount++; shieldedSpendsSpentCount++;
totalSpends += CAmount(-note.value()); totalSpends += CAmount(-note.value());
obj.push_back(Pair("address",EncodePaymentAddress(address))); obj.push_back(Pair("address",EncodePaymentAddress(address)));
@@ -197,16 +246,13 @@ void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpen
} }
void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, ZsTxJSONContext* ctx) { void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys) {
if(fZdebug) if(fZdebug)
fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() ); fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
LOCK2(cs_main, pwalletMain->cs_wallet); LOCK2(cs_main, pwalletMain->cs_wallet);
ZsTxJSONContext localCtx; // used only when the caller passed no per-call memo
ZsTxJSONContext& zctx = ctx ? *ctx : localCtx;
//Check address //Check address
bool isTAddress = false; bool isTAddress = false;
bool isZsAddress = false; bool isZsAddress = false;
@@ -256,45 +302,18 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
UniValue zsReceived(UniValue::VARR); UniValue zsReceived(UniValue::VARR);
if (isZsAddress || !filterByAddress) { if (isZsAddress || !filterByAddress) {
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) { for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
UniValue obj(UniValue::VOBJ); UniValue obj(UniValue::VOBJ);
bool changeTx = false; bool changeTx = false;
//Decrypt sapling incoming commitments using IVK //Decrypt sapling incoming commitments using IVK: once per key, not once per address
const std::vector<ZsTxJSONContext::Key>& keys = zctx.keys(); OutputDecryptions decryptions(wtx, i);
// An output opens under at most one ivk (decrypt() checks the note commitment, which binds for (const SaplingHistoryKeys::SpendableAddress& a : keys.spendable) {
// pk_d = [ivk]g_d), and FindMySaplingNotes records that ivk in mapSaplingNoteData. When the recorded const libzcash::SaplingPaymentAddress& addr = a.addr;
// ivk is one of keys(), pass 0 tries only the keys with that ivk: every other key would fail, as in {
// the old single loop. If pass 0 opens nothing (stale note data), pass 1 tries the remaining keys in const SaplingNoteCache* pt = decryptions.Get(a.ivk);
// the same order. Without a usable record only pass 1 runs, over every key, exactly as before.
libzcash::SaplingIncomingViewingKey recordedIvk;
bool narrow = false;
{
mapSaplingNoteData_t::const_iterator ndIt = wtx.mapSaplingNoteData.find(SaplingOutPoint(wtx.GetHash(), i));
if (ndIt != wtx.mapSaplingNoteData.end()) {
for (size_t k = 0; k < keys.size() && !narrow; k++) {
if (keys[k].ivk == ndIt->second.ivk) {
recordedIvk = ndIt->second.ivk;
narrow = true;
}
}
}
}
bool openedByRecordedIvk = false;
for (int pass = narrow ? 0 : 1; pass < 2 && !openedByRecordedIvk; pass++) {
for (size_t k = 0; k < keys.size(); k++) {
if (narrow && ((pass == 0) != (keys[k].ivk == recordedIvk)))
continue; // pass 0: only the recorded ivk; pass 1: only the others
const libzcash::SaplingPaymentAddress& addr = keys[k].addr;
{ // spending-key availability is resolved once per call, in ZsTxJSONContext::keys()
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext, keys[k].ivk, outputDesc.ephemeralKey, outputDesc.cm);
if (pt && pass == 0)
openedByRecordedIvk = true;
if (pt) { if (pt) {
auto note = pt.get(); const libzcash::SaplingNote& note = pt->note;
auto pt_unwrapped = pt.get(); auto memo = pt->memo;
auto memo = pt_unwrapped.memo();
obj.push_back(Pair("address",EncodePaymentAddress(addr))); obj.push_back(Pair("address",EncodePaymentAddress(addr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value())))); obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value()))));
obj.push_back(Pair("memo", HexStr(memo))); obj.push_back(Pair("memo", HexStr(memo)));
@@ -311,7 +330,7 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
//Check Change Status //Check Change Status
if (wtx.vShieldedSpend.size()!=0) { if (wtx.vShieldedSpend.size()!=0) {
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) { BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
if (zctx.hasNullifier(addr, spendDesc.nullifier)) { if (keys.IsNullifierOfAddress(a.walletIvk, spendDesc.nullifier)) {
changeTx = true; changeTx = true;
} }
} }
@@ -324,7 +343,6 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
} }
} }
} }
} // pass
} }
} }
received.push_back(Pair("saplingReceived",zsReceived)); received.push_back(Pair("saplingReceived",zsReceived));
@@ -333,13 +351,10 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
} }
void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, ZsTxJSONContext* ctx) { void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys) {
LOCK2(cs_main, pwalletMain->cs_wallet); LOCK2(cs_main, pwalletMain->cs_wallet);
ZsTxJSONContext localCtx; // used only when the caller passed no per-call memo
ZsTxJSONContext& zctx = ctx ? *ctx : localCtx;
//Used to identify incomplete key sets //Used to identify incomplete key sets
int shieldedOutputCount = 0; int shieldedOutputCount = 0;
int shieldedOutputDecryptedCount = 0; int shieldedOutputDecryptedCount = 0;
@@ -389,6 +404,13 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
//All Shielded Sends in the transaction //All Shielded Sends in the transaction
UniValue zsSends(UniValue::VARR); UniValue zsSends(UniValue::VARR);
if (isZsAddress || !filterByAddress) { if (isZsAddress || !filterByAddress) {
//The t to z shielding ovk comes from the HD seed: derive it once per tx, not once per output
boost::optional<uint256> seedOvk;
if (wtx.vShieldedSpend.size()==0 && wtx.vShieldedOutput.size()!=0) {
HDSeed seed;
if (pwalletMain->GetHDSeedForDerivation(seed))
seedOvk = ovkForShieldingFromTaddr(seed);
}
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) { for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
const OutputDescription& outputDesc = wtx.vShieldedOutput[i]; const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
shieldedOutputCount++; shieldedOutputCount++;
@@ -397,20 +419,14 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
//Decrypt sapling outgoing t to z transaction using HDseed //Decrypt sapling outgoing t to z transaction using HDseed
if (wtx.vShieldedSpend.size()==0) { if (wtx.vShieldedSpend.size()==0) {
uint256 taddrOvk; if (seedOvk) {
if (zctx.seedOvk(taddrOvk)) { {
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt( auto sent = DecryptSent(outputDesc, *seedOvk);
outputDesc.outCiphertext,taddrOvk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
if (opt) { if (sent) {
auto opt_unwrapped = opt.get();
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
if (pt) {
shieldedOutputDecryptedCount++; shieldedOutputDecryptedCount++;
auto pt_unwrapped = pt.get(); const libzcash::SaplingNotePlaintext& pt_unwrapped = sent->first;
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d); const libzcash::SaplingPaymentAddress& sentAddr = sent->second;
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr))); obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value())))); obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value()))));
obj.push_back(Pair("shieldedOutputIndex",i)); obj.push_back(Pair("shieldedOutputIndex",i));
@@ -423,24 +439,22 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
} }
} }
//attempt Decryption of Outgoing Sapling using wallet extended spending keys //attempt Decryption of Outgoing Sapling using wallet extended spending keys: once per ovk, not once per address
} else { } else {
const std::vector<ZsTxJSONContext::Key>& keys = zctx.keys(); std::map<uint256, boost::optional<std::pair<libzcash::SaplingNotePlaintext, libzcash::SaplingPaymentAddress>>> sentByOvk;
for (size_t k = 0; k < keys.size(); k++) { for (const SaplingHistoryKeys::SpendableAddress& a : keys.spendable) {
{ // spending-key availability is resolved once per call, in ZsTxJSONContext::keys() {
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt( auto tried = sentByOvk.find(a.ovk);
outputDesc.outCiphertext,keys[k].ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey); if (tried == sentByOvk.end())
tried = sentByOvk.insert(std::make_pair(a.ovk, DecryptSent(outputDesc, a.ovk))).first;
const auto& sent = tried->second;
if (opt) { {
auto opt_unwrapped = opt.get(); if (sent) {
auto pt = libzcash::SaplingNotePlaintext::decrypt( const libzcash::SaplingNotePlaintext& pt_unwrapped = sent->first;
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
if (pt) {
auto pt_unwrapped = pt.get();
auto memo = pt_unwrapped.memo(); auto memo = pt_unwrapped.memo();
shieldedOutputDecryptedCount++; shieldedOutputDecryptedCount++;
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d); const libzcash::SaplingPaymentAddress& sentAddr = sent->second;
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr))); obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value())))); obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value()))));
obj.push_back(Pair("memo", HexStr(memo))); obj.push_back(Pair("memo", HexStr(memo)));
@@ -456,8 +470,10 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
//Check Change Status //Check Change Status
if (wtx.vShieldedSpend.size()!=0) { if (wtx.vShieldedSpend.size()!=0) {
libzcash::SaplingIncomingViewingKey sentIvk;
pwalletMain->GetSaplingIncomingViewingKey(sentAddr, sentIvk);
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) { BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
if (zctx.hasNullifier(sentAddr, spendDesc.nullifier)) { if (keys.IsNullifierOfAddress(sentIvk, spendDesc.nullifier)) {
changeTx = true; changeTx = true;
} }
} }
@@ -486,7 +502,7 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
} }
void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, ZsTxJSONContext* ctx) { void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, const SaplingHistoryKeys& keys) {
LOCK2(cs_main, pwalletMain->cs_wallet); LOCK2(cs_main, pwalletMain->cs_wallet);
@@ -534,7 +550,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
tx.push_back(Pair("expiryheight", (int64_t)wtx.nExpiryHeight)); tx.push_back(Pair("expiryheight", (int64_t)wtx.nExpiryHeight));
tx.push_back(Pair("confirmations", confirms)); tx.push_back(Pair("confirmations", confirms));
tx.push_back(Pair("time", wtx.GetTxTime())); tx.push_back(Pair("time", wtx.GetTxTime()));
tx.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<const CTransaction&>(wtx), SER_NETWORK, PROTOCOL_VERSION)))); tx.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
//Wallet Conflicts //Wallet Conflicts
UniValue conflicts(UniValue::VARR); UniValue conflicts(UniValue::VARR);
@@ -558,7 +574,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
} }
// Get Received // Get Received
if (returnType == 0 || returnType == 2) { if (returnType == 0 || returnType == 2) {
zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool, ctx); zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool, keys);
if (!fBool || totalReceived != 0) { if (!fBool || totalReceived != 0) {
tx.push_back(Pair("received",received)); tx.push_back(Pair("received",received));
} }
@@ -568,7 +584,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
if (returnType == 0 || returnType == 3) { if (returnType == 0 || returnType == 3) {
//Only include sends if there are spends that belong to the wallet. //Only include sends if there are spends that belong to the wallet.
if (totalSpends != 0 || fBool) { if (totalSpends != 0 || fBool) {
zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool, ctx); zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool, keys);
} }
if (!fBool || totalSends != 0) { if (!fBool || totalSends != 0) {
tx.push_back(Pair("sends",sends)); tx.push_back(Pair("sends",sends));
src/wallet/rpchushwallet.h
+23 -41
View File
@@ -6,13 +6,6 @@
#ifndef HUSH_WALLET_RPCHUSHWALLET_H #ifndef HUSH_WALLET_RPCHUSHWALLET_H
#define HUSH_WALLET_RPCHUSHWALLET_H #define HUSH_WALLET_RPCHUSHWALLET_H
#include "uint256.h"
#include "zcash/Address.hpp"
#include <set>
#include <utility>
#include <vector>
struct balancestruct { struct balancestruct {
CAmount confirmed; CAmount confirmed;
CAmount unconfirmed; CAmount unconfirmed;
@@ -20,46 +13,35 @@ struct balancestruct {
CAmount immature; CAmount immature;
}; };
// Per-RPC-call memo for the zs*ToJSON helpers: the wallet's usable Sapling keys, the HD-seed ovk and a /** The wallet's Sapling keys as the history helpers use them, gathered once per RPC call. The helpers used to
// nullifier index, derived once per call instead of once per output. Every member is a copy; nothing * look them up for every output and every address, re-deriving each full viewing key (an elliptic-curve
// points into wallet maps. Valid only while the caller holds LOCK2(cs_main, pwalletMain->cs_wallet) for * operation), and walked every note in the wallet for each change check. Build it with pwalletMain->cs_wallet
// the whole call: declare it after that lock and never keep it across calls. Passing NULL to the helpers * held, and keep holding it while the helpers run: that keeps the wallet's keys and notes fixed. The one thing
// gives each helper call its own memo (the old per-call cost profile, minus the per-output repeats). * it can't stop is the walletpassphrase timer relocking the wallet mid-call; the call then reports the keys as
class ZsTxJSONContext { * they were when it started, where the old per-output lookups switched over partway. */
public: struct SaplingHistoryKeys {
struct Key { struct SpendableAddress {
libzcash::SaplingPaymentAddress addr; libzcash::SaplingPaymentAddress addr;
libzcash::SaplingIncomingViewingKey ivk; libzcash::SaplingIncomingViewingKey ivk; //!< from its spending key: what received outputs are decrypted with
uint256 ovk; uint256 ovk; //!< from its spending key: what outCiphertexts are decrypted with
libzcash::SaplingIncomingViewingKey walletIvk; //!< GetSaplingIncomingViewingKey(addr), as the change check looked it up
}; };
//! Every wallet address whose spending key is available (never while the wallet is locked), in
ZsTxJSONContext() : keysBuilt(false), nullifiersBuilt(false), seedOvkTried(false), seedOvkOk(false) {} //! GetSaplingPaymentAddresses order.
ZsTxJSONContext(const ZsTxJSONContext&) = delete; std::vector<SpendableAddress> spendable;
ZsTxJSONContext& operator=(const ZsTxJSONContext&) = delete; //! Whether GetNullifiersForAddresses({addr}) would contain (addr, nullifier), given addr's
//! GetSaplingIncomingViewingKey result (zero when the wallet doesn't know addr): nullifier belongs to a
// Wallet z-addresses (std::set order) whose extended spending key is available now, with that key's //! wallet note received with that ivk. Walks the wallet's notes once, on first use.
// extsk.expsk.full_viewing_key().in_viewing_key() and .ovk. Empty for an encrypted+locked wallet or bool IsNullifierOfAddress(const libzcash::SaplingIncomingViewingKey& addrIvk, const uint256& nullifier) const;
// one that holds only viewing keys.
const std::vector<Key>& keys();
// == GetNullifiersForAddresses({addr}).count(std::make_pair(addr, nullifier)) > 0
bool hasNullifier(const libzcash::SaplingPaymentAddress& addr, const uint256& nullifier);
// == GetHDSeedForDerivation(seed) ? (ovkOut = ovkForShieldingFromTaddr(seed), true) : false
bool seedOvk(uint256& ovkOut);
private: private:
bool keysBuilt; mutable boost::optional<std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256>>> noteNullifiers;
std::vector<Key> vKeys;
bool nullifiersBuilt;
std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256> > nfByIvk;
bool seedOvkTried;
bool seedOvkOk;
uint256 seedOvkValue;
}; };
SaplingHistoryKeys GetSaplingHistoryKeys();
void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress); void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress);
void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, ZsTxJSONContext* ctx = NULL); void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys);
void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, ZsTxJSONContext* ctx = NULL); void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys);
void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, ZsTxJSONContext* ctx = NULL); void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, const SaplingHistoryKeys& keys);
#endif //HUSH_WALLET_RPCWALLET_H #endif //HUSH_WALLET_RPCWALLET_H
src/wallet/rpcwallet.cpp
+104 -179
View File
@@ -48,7 +48,6 @@
#include <boost/assign/list_of.hpp> #include <boost/assign/list_of.hpp>
#include <utf8.h> #include <utf8.h>
#include <univalue.h> #include <univalue.h>
#include <algorithm>
#include <numeric> #include <numeric>
#include "hush_defs.h" #include "hush_defs.h"
#include <string.h> #include <string.h>
@@ -1774,19 +1773,7 @@ static void MaybePushAddress(UniValue & entry, const CTxDestination &dest)
} }
} }
// listtransactions skips its first `from` entries without building them: while *pnSkip > 0, an void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDepth, bool fLong, UniValue& ret, const isminefilter& filter)
// entry that would be emitted only decrements *pnSkip. The counter is shared across calls, so the
// skipped entries are exactly the first `from` ones in emission order. NULL: emit everything.
static bool ConsumeSkip(int* pnSkip)
{
if (pnSkip != NULL && *pnSkip > 0) {
--*pnSkip;
return true;
}
return false;
}
void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDepth, bool fLong, UniValue& ret, const isminefilter& filter, int* pnSkip = NULL)
{ {
CAmount nFee; CAmount nFee;
string strSentAccount; string strSentAccount;
@@ -1800,18 +1787,11 @@ void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDe
bool fAllAccounts = (strAccount == string("*")); bool fAllAccounts = (strAccount == string("*"));
bool involvesWatchonly = wtx.IsFromMe(ISMINE_WATCH_ONLY); bool involvesWatchonly = wtx.IsFromMe(ISMINE_WATCH_ONLY);
// "size" is a property of the transaction, not of an entry: compute it once, when the first
// entry is built, through a const reference instead of a by-value CTransaction copy.
bool fHaveTxSize = false;
uint64_t nTxSize = 0;
// Sent // Sent
if ((!listSent.empty() || nFee != 0) && (fAllAccounts || strAccount == strSentAccount)) if ((!listSent.empty() || nFee != 0) && (fAllAccounts || strAccount == strSentAccount))
{ {
BOOST_FOREACH(const COutputEntry& s, listSent) BOOST_FOREACH(const COutputEntry& s, listSent)
{ {
if (ConsumeSkip(pnSkip))
continue;
UniValue entry(UniValue::VOBJ); UniValue entry(UniValue::VOBJ);
if(involvesWatchonly || (::IsMine(*pwalletMain, s.destination) & ISMINE_WATCH_ONLY)) if(involvesWatchonly || (::IsMine(*pwalletMain, s.destination) & ISMINE_WATCH_ONLY))
entry.push_back(Pair("involvesWatchonly", true)); entry.push_back(Pair("involvesWatchonly", true));
@@ -1823,11 +1803,7 @@ void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDe
entry.push_back(Pair("fee", ValueFromAmount(-nFee))); entry.push_back(Pair("fee", ValueFromAmount(-nFee)));
if (fLong) if (fLong)
WalletTxToJSON(wtx, entry); WalletTxToJSON(wtx, entry);
if (!fHaveTxSize) { entry.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
nTxSize = static_cast<uint64_t>(GetSerializeSize(static_cast<const CTransaction&>(wtx), SER_NETWORK, PROTOCOL_VERSION));
fHaveTxSize = true;
}
entry.push_back(Pair("size", nTxSize));
ret.push_back(entry); ret.push_back(entry);
} }
} }
@@ -1842,8 +1818,6 @@ void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDe
account = pwalletMain->mapAddressBook[r.destination].name; account = pwalletMain->mapAddressBook[r.destination].name;
if (fAllAccounts || (account == strAccount)) if (fAllAccounts || (account == strAccount))
{ {
if (ConsumeSkip(pnSkip))
continue;
UniValue entry(UniValue::VOBJ); UniValue entry(UniValue::VOBJ);
if(involvesWatchonly || (::IsMine(*pwalletMain, r.destination) & ISMINE_WATCH_ONLY)) if(involvesWatchonly || (::IsMine(*pwalletMain, r.destination) & ISMINE_WATCH_ONLY))
entry.push_back(Pair("involvesWatchonly", true)); entry.push_back(Pair("involvesWatchonly", true));
@@ -1874,25 +1848,19 @@ void ListTransactions(const CWalletTx& wtx, const string& strAccount, int nMinDe
entry.push_back(Pair("vout", r.vout)); entry.push_back(Pair("vout", r.vout));
if (fLong) if (fLong)
WalletTxToJSON(wtx, entry); WalletTxToJSON(wtx, entry);
if (!fHaveTxSize) { entry.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
nTxSize = static_cast<uint64_t>(GetSerializeSize(static_cast<const CTransaction&>(wtx), SER_NETWORK, PROTOCOL_VERSION));
fHaveTxSize = true;
}
entry.push_back(Pair("size", nTxSize));
ret.push_back(entry); ret.push_back(entry);
} }
} }
} }
} }
void AcentryToJSON(const CAccountingEntry& acentry, const string& strAccount, UniValue& ret, int* pnSkip = NULL) void AcentryToJSON(const CAccountingEntry& acentry, const string& strAccount, UniValue& ret)
{ {
bool fAllAccounts = (strAccount == string("*")); bool fAllAccounts = (strAccount == string("*"));
if (fAllAccounts || acentry.strAccount == strAccount) if (fAllAccounts || acentry.strAccount == strAccount)
{ {
if (ConsumeSkip(pnSkip))
return;
UniValue entry(UniValue::VOBJ); UniValue entry(UniValue::VOBJ);
entry.push_back(Pair("account", acentry.strAccount)); entry.push_back(Pair("account", acentry.strAccount));
entry.push_back(Pair("category", "move")); entry.push_back(Pair("category", "move"));
@@ -1986,42 +1954,46 @@ UniValue listtransactions(const UniValue& params, bool fHelp, const CPubKey& myp
UniValue ret(UniValue::VARR); UniValue ret(UniValue::VARR);
std::list<CAccountingEntry> acentries; std::list<CAccountingEntry> acentries;
CWallet::TxItems txOrdered = pwalletMain->OrderedTxItems(acentries, strAccount, true); CWallet::TxItems txOrdered = pwalletMain->OrderedTxItems(acentries, strAccount);
// iterate backwards until we have nCount items to return. The first nFrom entries are only // iterate backwards until we have nCount items to return:
// counted (nSkipLeft), never built, since they are not returned.
int nSkipLeft = nFrom;
for (CWallet::TxItems::reverse_iterator it = txOrdered.rbegin(); it != txOrdered.rend(); ++it) for (CWallet::TxItems::reverse_iterator it = txOrdered.rbegin(); it != txOrdered.rend(); ++it)
{ {
CWalletTx *const pwtx = (*it).second.first; CWalletTx *const pwtx = (*it).second.first;
if (pwtx != 0) if (pwtx != 0)
{ {
ListTransactions(*pwtx, strAccount, 0, true, ret, filter, &nSkipLeft); ListTransactions(*pwtx, strAccount, 0, true, ret, filter);
} }
CAccountingEntry *const pacentry = (*it).second.second; CAccountingEntry *const pacentry = (*it).second.second;
if (pacentry != 0) if (pacentry != 0)
AcentryToJSON(*pacentry, strAccount, ret, &nSkipLeft); AcentryToJSON(*pacentry, strAccount, ret);
// (skipped + built) is the entry count the loop used to test if ((int)ret.size() >= (nCount+nFrom)) break;
if ((nFrom - nSkipLeft) + (int)ret.size() >= (nCount+nFrom)) break;
} }
// ret holds the entries after the first nSkipped ones, newest to oldest // ret is newest to oldest
const int nSkipped = nFrom - nSkipLeft;
const int nSeen = nSkipped + (int)ret.size();
if (nFrom > nSeen) if (nFrom > (int)ret.size())
nFrom = nSeen; nFrom = ret.size();
if ((nFrom + nCount) > nSeen) if ((nFrom + nCount) > (int)ret.size())
nCount = nSeen - nFrom; nCount = ret.size() - nFrom;
// The reply is entries [nFrom, nFrom+nCount) of the newest-first sequence, oldest to newest. vector<UniValue> arrTmp = ret.getValues();
// Entry k of that sequence is ret[k - nSkipped] (nFrom >= nSkipped holds here).
UniValue result(UniValue::VARR);
const std::vector<UniValue>& vNewestFirst = ret.getValues();
for (int i = nFrom + nCount; i > nFrom; --i)
result.push_back(vNewestFirst[i - 1 - nSkipped]);
return result; vector<UniValue>::iterator first = arrTmp.begin();
std::advance(first, nFrom);
vector<UniValue>::iterator last = arrTmp.begin();
std::advance(last, nFrom+nCount);
if (last != arrTmp.end()) arrTmp.erase(last, arrTmp.end());
if (first != arrTmp.begin()) arrTmp.erase(arrTmp.begin(), first);
std::reverse(arrTmp.begin(), arrTmp.end()); // Return oldest to newest
ret.clear();
ret.setArray();
ret.push_backV(arrTmp);
return ret;
} }
UniValue listaccounts(const UniValue& params, bool fHelp, const CPubKey& mypk) UniValue listaccounts(const UniValue& params, bool fHelp, const CPubKey& mypk)
@@ -2266,7 +2238,7 @@ UniValue gettransaction(const UniValue& params, bool fHelp, const CPubKey& mypk)
ListTransactions(wtx, "*", 0, false, details, filter); ListTransactions(wtx, "*", 0, false, details, filter);
entry.push_back(Pair("details", details)); entry.push_back(Pair("details", details));
string strHex = EncodeHexTx(static_cast<const CTransaction&>(wtx)); string strHex = EncodeHexTx(static_cast<CTransaction>(wtx));
entry.push_back(Pair("hex", strHex)); entry.push_back(Pair("hex", strHex));
return entry; return entry;
@@ -2988,26 +2960,6 @@ UniValue listunspent(const UniValue& params, bool fHelp, const CPubKey& mypk)
return results; return results;
} }
// Wallet transactions ordered by nOrderPos without copying them (the old code copied every CWalletTx,
// with every note's witness list, into a map<int64_t,CWalletTx>). The pointers point into
// pwalletMain->mapWallet and are valid only while cs_wallet is held and nothing erases from mapWallet
// (std::map insertions do not invalidate them). Reproduces map<int64_t,CWalletTx>::insert exactly:
// ascending nOrderPos and, for equal nOrderPos, only the first tx in mapWallet (txid) order, because
// insert() never replaces an existing key; hence stable_sort + unique.
typedef std::pair<int64_t, const CWalletTx*> OrderedWtx;
static bool OrderedWtxLess(const OrderedWtx& a, const OrderedWtx& b) { return a.first < b.first; }
static bool OrderedWtxSamePos(const OrderedWtx& a, const OrderedWtx& b) { return a.first == b.first; }
static void BuildOrderedWalletTxs(std::vector<OrderedWtx>& out)
{
AssertLockHeld(pwalletMain->cs_wallet);
out.clear();
out.reserve(pwalletMain->mapWallet.size());
for (std::map<uint256, CWalletTx>::const_iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it)
out.push_back(OrderedWtx(it->second.nOrderPos, &it->second));
std::stable_sort(out.begin(), out.end(), OrderedWtxLess);
out.erase(std::unique(out.begin(), out.end(), OrderedWtxSamePos), out.end());
}
// Return current blockchain status, wallet balance, address balance and the last 200 transactions // Return current blockchain status, wallet balance, address balance and the last 200 transactions
UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&) UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
{ {
@@ -3079,14 +3031,14 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
addressBalances.insert(make_pair(addressString,txAmounts)); addressBalances.insert(make_pair(addressString,txAmounts));
} }
// Per-call key/nullifier memo for the note loop below and for zsWalletTxJSON (rpchushwallet.h). //Create Ordered List
// Declared after the LOCK2 above, so it is destroyed before cs_wallet is released. // Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
ZsTxJSONContext zsCtx; // As before, the first tx inserted for an nOrderPos wins.
map<int64_t,const CWalletTx*> orderedTxs;
//Classify every wallet tx for the balances (the ordered list for listtransactions is built below)
for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) { for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
const uint256& wtxid = it->first; const uint256& wtxid = it->first;
const CWalletTx& wtx = (*it).second; const CWalletTx& wtx = (*it).second;
orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
unsigned int txType = 0; unsigned int txType = 0;
// 0 Unassigend // 0 Unassigend
@@ -3155,19 +3107,18 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
for (auto & pair : wtx.mapSaplingNoteData) { for (auto & pair : wtx.mapSaplingNoteData) {
SaplingOutPoint op = pair.first; SaplingOutPoint op = pair.first;
const SaplingNoteData& nd = pair.second; SaplingNoteData nd = pair.second;
//Skip Spent //Skip Spent
if (nd.nullifier && pwalletMain->IsSaplingSpent(*nd.nullifier)) if (nd.nullifier && pwalletMain->IsSaplingSpent(*nd.nullifier))
continue; continue;
//Decrypt sapling incoming commitments using IVK //Decrypt sapling incoming commitments using IVK
const std::vector<ZsTxJSONContext::Key>& zsKeys = zsCtx.keys(); for (auto addr : zs_addresses) {
for (size_t k = 0; k < zsKeys.size(); k++) { libzcash::SaplingExtendedSpendingKey extsk;
const libzcash::SaplingPaymentAddress& addr = zsKeys[k].addr; if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
{ // spending-key availability is resolved once per call, in ZsTxJSONContext::keys()
auto pt = libzcash::SaplingNotePlaintext::decrypt( auto pt = libzcash::SaplingNotePlaintext::decrypt(
wtx.vShieldedOutput[op.n].encCiphertext,zsKeys[k].ivk,wtx.vShieldedOutput[op.n].ephemeralKey,wtx.vShieldedOutput[op.n].cm); wtx.vShieldedOutput[op.n].encCiphertext,extsk.expsk.full_viewing_key().in_viewing_key(),wtx.vShieldedOutput[op.n].ephemeralKey,wtx.vShieldedOutput[op.n].cm);
if (txType == 0 && pwalletMain->IsLockedNote(op)) if (txType == 0 && pwalletMain->IsLockedNote(op))
txType == 3; txType == 3;
@@ -3279,14 +3230,11 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
} }
//Ordered list of pointers into mapWallet (cs_wallet is held for the whole call) const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
std::vector<OrderedWtx> orderedTxs;
BuildOrderedWalletTxs(orderedTxs);
uint64_t t = GetTime(); uint64_t t = GetTime();
for (std::vector<OrderedWtx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it)
{ {
const CWalletTx& wtx = *(it->second); const CWalletTx& wtx = *(*it).second;
if (!CheckFinalTx(wtx)) if (!CheckFinalTx(wtx))
continue; continue;
@@ -3303,7 +3251,7 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
continue; continue;
} }
zsWalletTxJSON(wtx, trans, "*", false, 0, &zsCtx); zsWalletTxJSON(wtx, trans, "*", false, 0, historyKeys);
if (trans.size() >= nCount) break; if (trans.size() >= nCount) break;
} }
@@ -3569,16 +3517,21 @@ UniValue z_listreceivedaddress(const UniValue& params, bool fHelp,const CPubKey&
if (nFilter < 0) if (nFilter < 0)
throw runtime_error("Filter must be greater that 0."); throw runtime_error("Filter must be greater that 0.");
//Ordered list of pointers into mapWallet (cs_wallet is held for the whole call) //Created Ordered Transaction Map
std::vector<OrderedWtx> orderedTxs; // Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
BuildOrderedWalletTxs(orderedTxs); // As before, the first tx inserted for an nOrderPos wins.
ZsTxJSONContext zsCtx; // per-call key/nullifier memo (rpchushwallet.h) map<int64_t,const CWalletTx*> orderedTxs;
for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
const CWalletTx& wtx = (*it).second;
orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
}
const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
uint64_t t = GetTime(); uint64_t t = GetTime();
//Reverse Iterate thru transactions //Reverse Iterate thru transactions
for (std::vector<OrderedWtx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) { for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
const CWalletTx& wtx = *(it->second); const CWalletTx& wtx = *(*it).second;
bool includeTransaction = true; bool includeTransaction = true;
//Excude transactions with less confirmations than required //Excude transactions with less confirmations than required
@@ -3599,7 +3552,7 @@ UniValue z_listreceivedaddress(const UniValue& params, bool fHelp,const CPubKey&
} }
if (includeTransaction) { if (includeTransaction) {
zsWalletTxJSON(wtx, ret, params[0].get_str() , true, 2, &zsCtx); zsWalletTxJSON(wtx, ret, params[0].get_str() , true, 2, historyKeys);
} }
if (ret.size() > nCount) break; if (ret.size() > nCount) break;
@@ -3650,7 +3603,7 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
if (!EnsureWalletIsAvailable(fHelp)) if (!EnsureWalletIsAvailable(fHelp))
return NullUniValue; return NullUniValue;
if (fHelp || params.size() > 6 || params.size() == 3) if (fHelp || params.size() > 5 || params.size() == 3)
throw runtime_error( throw runtime_error(
"z_listsentbyaddress\n" "z_listsentbyaddress\n"
"\nReturns decrypted DragonX outputs sent to a single address.\n" "\nReturns decrypted DragonX outputs sent to a single address.\n"
@@ -3675,16 +3628,11 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
"5. \"Count:\" (numeric, optional, default=9999999) \n" "5. \"Count:\" (numeric, optional, default=9999999) \n"
" Last n number of transactions returned\n" " Last n number of transactions returned\n"
"\n" "\n"
"6. \"Sends only:\" (boolean, optional, default=false) \n"
" true: omit the \"spends\" and \"received\" keys of every entry and skip the\n"
" trial decryption they need. false: the full historic output.\n"
"\n"
"Default Parameters:\n" "Default Parameters:\n"
"2. 0 - O confimations required\n" "2. 0 - O confimations required\n"
"3. 0 - Returns all transactions\n" "3. 0 - Returns all transactions\n"
"4. 9999999 - Ignored\n" "4. 9999999 - Ignored\n"
"5. 9999999 - Return the last 9,999,999 transactions.\n" "5. 9999999 - Return the last 9,999,999 transactions.\n"
"6. false - Include the spends and received sections.\n"
"\n" "\n"
"\nResult:\n" "\nResult:\n"
" \"txid\": \"transactionid\", (string) The transaction id.\n" " \"txid\": \"transactionid\", (string) The transaction id.\n"
@@ -3733,7 +3681,6 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
int64_t nFilterType = 0; int64_t nFilterType = 0;
int64_t nFilter = 9999999; int64_t nFilter = 9999999;
int64_t nCount = 9999999; int64_t nCount = 9999999;
bool fSendsOnly = false; // opt-in, see help: omits the "spends"/"received" keys
if (params.size() >= 2) if (params.size() >= 2)
nMinConfirms = params[1].get_int64(); nMinConfirms = params[1].get_int64();
@@ -3743,14 +3690,10 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
nFilter = params[3].get_int64(); nFilter = params[3].get_int64();
} }
if (params.size() >= 5) { if (params.size() == 5) {
nCount = params[4].get_int64(); nCount = params[4].get_int64();
} }
if (params.size() == 6) {
fSendsOnly = params[5].get_bool();
}
if (nMinConfirms < 0) if (nMinConfirms < 0)
throw runtime_error("Minimum confimations must be greater that 0"); throw runtime_error("Minimum confimations must be greater that 0");
@@ -3760,16 +3703,21 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
if (nFilter < 0) if (nFilter < 0)
throw runtime_error("Filter must be greater that 0."); throw runtime_error("Filter must be greater that 0.");
//Ordered list of pointers into mapWallet (cs_wallet is held for the whole call) //Created Ordered Transaction Map
std::vector<OrderedWtx> orderedTxs; // Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
BuildOrderedWalletTxs(orderedTxs); // As before, the first tx inserted for an nOrderPos wins.
ZsTxJSONContext zsCtx; // per-call key/nullifier memo (rpchushwallet.h) map<int64_t,const CWalletTx*> orderedTxs;
for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
const CWalletTx& wtx = (*it).second;
orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
}
const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
uint64_t t = GetTime(); uint64_t t = GetTime();
//Reverse Iterate thru transactions //Reverse Iterate thru transactions
for (std::vector<OrderedWtx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) { for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
const CWalletTx& wtx = *(it->second); const CWalletTx& wtx = *(*it).second;
if (!CheckFinalTx(wtx)) if (!CheckFinalTx(wtx))
continue; continue;
@@ -3792,21 +3740,21 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
if (nFilterType == 2 && wtx.GetDepthInMainChain() > nFilter) if (nFilterType == 2 && wtx.GetDepthInMainChain() > nFilter)
continue; continue;
zsWalletTxJSON(wtx, ret, "*", false, fSendsOnly ? 3 : 0, &zsCtx); zsWalletTxJSON(wtx, ret, "*", false, 0, historyKeys);
if (ret.size() >= nCount) break; if (ret.size() >= nCount) break;
} }
// Return oldest to newest (ret was filled newest first): one copy of each entry instead of vector<UniValue> arrTmp = ret.getValues();
// getValues() + std::reverse + push_backV
UniValue oldestFirst(UniValue::VARR); std::reverse(arrTmp.begin(), arrTmp.end()); // Return oldest to newest
{
const std::vector<UniValue>& newestFirst = ret.getValues(); ret.clear();
for (std::vector<UniValue>::const_reverse_iterator rit = newestFirst.rbegin(); rit != newestFirst.rend(); ++rit) ret.setArray();
oldestFirst.push_back(*rit); ret.push_backV(arrTmp);
}
return oldestFirst; return ret;
} }
UniValue z_getstats(const UniValue& params, bool fHelp, const CPubKey& mypk) UniValue z_getstats(const UniValue& params, bool fHelp, const CPubKey& mypk)
@@ -4370,7 +4318,7 @@ UniValue z_listunspent(const UniValue& params, bool fHelp, const CPubKey& mypk)
for (auto & entry : saplingEntries) { for (auto & entry : saplingEntries) {
UniValue obj(UniValue::VOBJ); UniValue obj(UniValue::VOBJ);
int nHeight = tx_height(entry.op.hash); int nHeight = HeightFromDepth(entry.confirmations);
int dpowconfs = hush_dpowconfs(nHeight, entry.confirmations); int dpowconfs = hush_dpowconfs(nHeight, entry.confirmations);
// Only return notarized results when minconf>1 // Only return notarized results when minconf>1
@@ -4703,7 +4651,7 @@ UniValue z_listreceivedbyaddress(const UniValue& params, bool fHelp, const CPubK
if (boost::get<libzcash::SaplingPaymentAddress>(&zaddr) != nullptr) { if (boost::get<libzcash::SaplingPaymentAddress>(&zaddr) != nullptr) {
for (SaplingNoteEntry & entry : saplingEntries) { for (SaplingNoteEntry & entry : saplingEntries) {
UniValue obj(UniValue::VOBJ); UniValue obj(UniValue::VOBJ);
int nHeight = tx_height(entry.op.hash); int nHeight = HeightFromDepth(entry.confirmations);
int dpowconfs = hush_dpowconfs(nHeight, entry.confirmations); int dpowconfs = hush_dpowconfs(nHeight, entry.confirmations);
// Only return notarized results when minconf>1 // Only return notarized results when minconf>1
if (nMinDepth > 1 && dpowconfs == 1) if (nMinDepth > 1 && dpowconfs == 1)
@@ -4723,7 +4671,7 @@ UniValue z_listreceivedbyaddress(const UniValue& params, bool fHelp, const CPubK
} }
obj.push_back(Pair("outindex", (int)entry.op.n)); obj.push_back(Pair("outindex", (int)entry.op.n));
obj.push_back(Pair("rawconfirmations", entry.confirmations)); obj.push_back(Pair("rawconfirmations", entry.confirmations));
const CWalletTx& wtx = pwalletMain->mapWallet.at(entry.op.hash); //.ToString()); const CWalletTx& wtx = pwalletMain->mapWallet.at(entry.op.hash);
obj.push_back(Pair("time", wtx.GetTxTime())); obj.push_back(Pair("time", wtx.GetTxTime()));
obj.push_back(Pair("confirmations", dpowconfs)); obj.push_back(Pair("confirmations", dpowconfs));
@@ -4976,7 +4924,7 @@ UniValue z_viewtransaction(const UniValue& params, bool fHelp, const CPubKey& my
continue; continue;
} }
auto op = res->second; auto op = res->second;
const CWalletTx& wtxPrev = pwalletMain->mapWallet.at(op.hash); auto wtxPrev = pwalletMain->mapWallet.at(op.hash);
auto decrypted = wtxPrev.DecryptSaplingNote(op).get(); auto decrypted = wtxPrev.DecryptSaplingNote(op).get();
auto notePt = decrypted.first; auto notePt = decrypted.first;
auto pa = decrypted.second; auto pa = decrypted.second;
@@ -5079,12 +5027,7 @@ UniValue z_getoperationresult(const UniValue& params, bool fHelp, const CPubKey&
+ HelpExampleRpc("z_getoperationresult", "'[\"operationid\", ... ]'") + HelpExampleRpc("z_getoperationresult", "'[\"operationid\", ... ]'")
); );
// This call will remove finished operations. Unlike z_getoperationstatus it keeps cs_main + // This call will remove finished operations
// cs_wallet: CWallet::RunSaplingSweep/RunSaplingConsolidation/RunAutoShieldCoinbase cancel and
// remove the previous automated operation inside one cs_wallet critical section, and holding
// cs_wallet here keeps this removal serialised with that and with other z_getoperationresult
// callers, exactly as before.
LOCK2(cs_main, pwalletMain->cs_wallet);
return z_getoperationstatus_IMPL(params, true); return z_getoperationstatus_IMPL(params, true);
} }
@@ -5113,12 +5056,7 @@ UniValue z_getoperationstatus(const UniValue& params, bool fHelp, const CPubKey&
UniValue z_getoperationstatus_IMPL(const UniValue& params, bool fRemoveFinishedOperations=false) UniValue z_getoperationstatus_IMPL(const UniValue& params, bool fRemoveFinishedOperations=false)
{ {
// No cs_main/cs_wallet here: nothing below reads wallet or chain state. AsyncRPCQueue guards LOCK2(cs_main, pwalletMain->cs_wallet);
// its operation map with its own mutex, and AsyncRPCOperation::getStatus() (and every
// override) reads only the atomic state_, members fixed at construction (id_, creation_time_,
// contextinfo_, targetHeight_) and result/error under the operation's own lock_. Holding the
// wallet locks only made this poll wait behind wallet scans and block connection. The removing
// form is serialised by its caller, z_getoperationresult, which still holds cs_main + cs_wallet.
std::set<AsyncRPCOperationId> filter; std::set<AsyncRPCOperationId> filter;
if (params.size()==1) { if (params.size()==1) {
@@ -5129,18 +5067,11 @@ UniValue z_getoperationstatus_IMPL(const UniValue& params, bool fRemoveFinishedO
} }
bool useFilter = (filter.size()>0); bool useFilter = (filter.size()>0);
UniValue ret(UniValue::VARR);
std::shared_ptr<AsyncRPCQueue> q = getAsyncRPCQueue(); std::shared_ptr<AsyncRPCQueue> q = getAsyncRPCQueue();
std::vector<AsyncRPCOperationId> ids = q->getAllOperationIds(); std::vector<AsyncRPCOperationId> ids = q->getAllOperationIds();
// Status objects in enumeration order, and one (creation_time, index) key per object for the for (auto id : ids) {
// sort. UniValue has no move operations, so sorting the objects themselves deep-copied them on
// every comparison and every element move.
std::vector<UniValue> objs;
objs.reserve(ids.size());
std::vector<std::pair<int64_t, size_t> > order;
order.reserve(ids.size());
for (const AsyncRPCOperationId& id : ids) {
if (useFilter && !filter.count(id)) if (useFilter && !filter.count(id))
continue; continue;
@@ -5156,32 +5087,26 @@ UniValue z_getoperationstatus_IMPL(const UniValue& params, bool fRemoveFinishedO
if (fRemoveFinishedOperations) { if (fRemoveFinishedOperations) {
// Caller is only interested in retrieving finished results // Caller is only interested in retrieving finished results
if ("success"==s || "failed"==s || "cancelled"==s) { if ("success"==s || "failed"==s || "cancelled"==s) {
objs.push_back(obj); ret.push_back(obj);
// Report the operation only if this call is the one that removed it. q->popOperationForId(id);
if (q->popOperationForId(id)) {
order.push_back(std::make_pair(operation->getCreationTime(), objs.size() - 1));
} else {
objs.pop_back();
}
} }
} else { } else {
objs.push_back(obj); ret.push_back(obj);
order.push_back(std::make_pair(operation->getCreationTime(), objs.size() - 1));
} }
} }
// sort results chronologically by creation_time. Compare creation_time only, never the index: std::vector<UniValue> arrTmp = ret.getValues();
// std::sort then makes exactly the comparisons and moves it made on the status objects before,
// so operations created in the same second keep the order earlier releases returned. // sort results chronologically by creation_time
std::sort(order.begin(), order.end(), std::sort(arrTmp.begin(), arrTmp.end(), [](UniValue a, UniValue b) -> bool {
[](const std::pair<int64_t, size_t>& a, const std::pair<int64_t, size_t>& b) -> bool { const int64_t t1 = find_value(a.get_obj(), "creation_time").get_int64();
return a.first < b.first; const int64_t t2 = find_value(b.get_obj(), "creation_time").get_int64();
return t1 < t2;
}); });
UniValue ret(UniValue::VARR); ret.clear();
for (size_t i = 0; i < order.size(); i++) { ret.setArray();
ret.push_back(objs[order[i].second]); ret.push_backV(arrTmp);
}
return ret; return ret;
} }
@@ -5460,7 +5385,7 @@ UniValue z_sendmany(const UniValue& params, bool fHelp, const CPubKey& mypk)
// sort notes from largest to smallest, which means // sort notes from largest to smallest, which means
// we will spend the largest first // we will spend the largest first
std::sort(saplingEntries.begin(), saplingEntries.end(), std::sort(saplingEntries.begin(), saplingEntries.end(),
[](SaplingNoteEntry i, SaplingNoteEntry j) -> bool { [](const SaplingNoteEntry& i, const SaplingNoteEntry& j) -> bool {
return i.note.value() > j.note.value(); return i.note.value() > j.note.value();
}); });
@@ -6268,8 +6193,8 @@ UniValue z_listoperationids(const UniValue& params, bool fHelp, const CPubKey& m
+ HelpExampleRpc("z_listoperationids", "") + HelpExampleRpc("z_listoperationids", "")
); );
// No cs_main/cs_wallet: like z_getoperationstatus this only reads the self-locking async LOCK2(cs_main, pwalletMain->cs_wallet);
// operation queue and each operation's atomic state.
std::string filter; std::string filter;
bool useFilter = false; bool useFilter = false;
if (params.size()==1) { if (params.size()==1) {
src/wallet/wallet.cpp
+412 -183
View File
@@ -43,7 +43,9 @@
#include "wallet/asyncrpcoperation_autoshieldcoinbase.h" #include "wallet/asyncrpcoperation_autoshieldcoinbase.h"
#include <random> #include <random>
#include <limits> #include <limits>
#include <system_error>
#include <thread> #include <thread>
#include <chrono>
#include <atomic> #include <atomic>
#include <mutex> #include <mutex>
#include "zcash/zip32.h" #include "zcash/zip32.h"
@@ -75,8 +77,8 @@ bool fPayAtLeastCustomFee = true;
CBlockIndex *hush_chainactive(int32_t height); CBlockIndex *hush_chainactive(int32_t height);
extern int32_t HUSH_LOADINGBLOCKS; extern int32_t HUSH_LOADINGBLOCKS;
int32_t hush_dpowconfs(int32_t height,int32_t numconfs); int32_t hush_dpowconfs(int32_t height,int32_t numconfs);
int tx_height( const uint256 &hash );
bool fTxDeleteEnabled = false; bool fTxDeleteEnabled = false;
bool fCheckNoteCache = false;
bool fTxConflictDeleteEnabled = false; bool fTxConflictDeleteEnabled = false;
int fDeleteInterval = DEFAULT_TX_DELETE_INTERVAL; int fDeleteInterval = DEFAULT_TX_DELETE_INTERVAL;
unsigned int fDeleteTransactionsAfterNBlocks = DEFAULT_TX_RETENTION_BLOCKS; unsigned int fDeleteTransactionsAfterNBlocks = DEFAULT_TX_RETENTION_BLOCKS;
@@ -575,6 +577,23 @@ void CWallet::ChainTip(const CBlockIndex *pindex,
DecrementNoteWitnesses(pindex); DecrementNoteWitnesses(pindex);
UpdateNullifierNoteMapForBlock(pblock); UpdateNullifierNoteMapForBlock(pblock);
} }
// Track the block the wallet's state reflects, advancing only along blocks delivered in order:
// a gap (the startup rescan was interrupted, so the wallet sits at its checkpoint behind the
// notifier) pins the pointer, and SetBestChain keeps stamping the checkpoint.
LOCK(cs_wallet);
if (added) {
if (pindexWalletTip == nullptr || pindex->pprev == pindexWalletTip)
pindexWalletTip = pindex;
} else if (pindex == pindexWalletTip) {
pindexWalletTip = pindex->pprev;
}
}
void CWallet::SetWalletTip(const CBlockIndex* pindex)
{
LOCK(cs_wallet);
pindexWalletTip = pindex;
} }
void CWallet::RunSaplingSweep(int blockHeight) { void CWallet::RunSaplingSweep(int blockHeight) {
@@ -801,10 +820,22 @@ bool CWallet::CommitAutomatedTx(const CTransaction& tx) {
void CWallet::SetBestChain(const CBlockLocator& loc) void CWallet::SetBestChain(const CBlockLocator& loc)
{ {
// FlushStateToDisk passes the chain tip, but ThreadNotifyWallets delivers blocks to the wallet
// asynchronously and can lag it. Stamping a locator ahead of what the wallet has processed makes
// the next start skip those blocks for good (no rescan, no notification): notes received in them
// go missing and spends in them go unseen. So stamp the block the wallet's state reflects (seeded
// by init before the notifier starts, then advanced by ChainTip). Init's own calls, made before
// it is seeded, pass the right locator themselves.
CBlockLocator locator = loc;
{
LOCK2(cs_main, cs_wallet);
if (pindexWalletTip != nullptr)
locator = chainActive.GetLocator(pindexWalletTip);
}
// Default ctor => fFlushOnClose=true => ~CDB runs a full BDB txn_checkpoint over the entire // Default ctor => fFlushOnClose=true => ~CDB runs a full BDB txn_checkpoint over the entire
// cache. Fine for the hourly/shutdown callers; ruinous inside a scan (see SetBestChainNoFlush). // cache. Fine for the hourly/shutdown callers; ruinous inside a scan (see SetBestChainNoFlush).
CWalletDB walletdb(strWalletFile); CWalletDB walletdb(strWalletFile);
SetBestChainINTERNAL(walletdb, loc); SetBestChainINTERNAL(walletdb, locator);
} }
bool CWallet::SetBestChainNoFlush(const CBlockLocator& loc) bool CWallet::SetBestChainNoFlush(const CBlockLocator& loc)
@@ -1232,9 +1263,11 @@ void ClearSingleNoteWitnessCache(NoteData* nd)
nd->witnessRootValidated = false; nd->witnessRootValidated = false;
} }
int CWallet::SaplingWitnessMinimumHeight(const uint256& nullifier, int nWitnessHeight, int nMinimumHeight) // A note without a nullifier is never extended by BuildWitnessCache (its pending-note gate requires one),
// so it doesn't pull the rebuild's start height down.
int CWallet::SaplingWitnessMinimumHeight(const boost::optional<uint256>& nullifier, int nWitnessHeight, int nMinimumHeight)
{ {
if (GetSaplingSpendDepth(nullifier) <= WITNESS_CACHE_SIZE) { if (nullifier && GetSaplingSpendDepth(*nullifier) <= WITNESS_CACHE_SIZE) {
nMinimumHeight = min(nWitnessHeight, nMinimumHeight); nMinimumHeight = min(nWitnessHeight, nMinimumHeight);
} }
return nMinimumHeight; return nMinimumHeight;
@@ -1266,15 +1299,16 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
uint256 blockRoot; uint256 blockRoot;
uint256 witnessRoot; uint256 witnessRoot;
if (!nd->nullifier) { //A note found with an incoming viewing key only (z_importviewingkey) can't be spent, so it needs
//no witness, and it never gets a nullifier: deriving one takes the full viewing key, which
//UpdateSaplingNullifierNoteMapWithTx skips without. Seeding it here meant clearing and reseeding
//it from disk on every call (the no-nullifier rule below), then aborting on the missing nullifier
//the first time it was confirmed. If its spending key is imported later, the rule below seeds it.
if (!HaveSaplingFullViewingKey(nd->ivk))
continue;
if (!nd->nullifier)
::ClearSingleNoteWitnessCache(nd); ::ClearSingleNoteWitnessCache(nd);
// A note found with only an incoming viewing key (z_importviewingkey) never gets a
// nullifier (UpdateSaplingNullifierNoteMapWithTx needs the full viewing key) and can't be
// spent, so it needs no witness. Building one re-read its block on every call and then
// aborted on *item.second.nullifier below.
if (!HaveSaplingFullViewingKey(nd->ivk))
continue;
}
if (!nd->witnesses.empty() && nd->witnessHeight > 0) { if (!nd->witnesses.empty() && nd->witnessHeight > 0) {
@@ -1284,7 +1318,7 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
//Skip Validation when witness root has been validated //Skip Validation when witness root has been validated
if (nd->witnessRootValidated) { if (nd->witnessRootValidated) {
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight); nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
continue; continue;
} }
@@ -1296,12 +1330,12 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
CBlockIndex* whIndex = chainActive[nd->witnessHeight]; CBlockIndex* whIndex = chainActive[nd->witnessHeight];
if (whIndex == NULL) { if (whIndex == NULL) {
//witnessHeight strictly above the active chain (transient catch-up): cannot validate yet //witnessHeight strictly above the active chain (transient catch-up): cannot validate yet
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight); nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
continue; continue;
} }
if (nd->witnesses.front().root() == whIndex->hashFinalSaplingRoot) { if (nd->witnesses.front().root() == whIndex->hashFinalSaplingRoot) {
nd->witnessRootValidated = true; nd->witnessRootValidated = true;
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight); nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
continue; continue;
} }
//root mismatch on the active chain -> desynced; fall through to rebuild below //root mismatch on the active chain -> desynced; fall through to rebuild below
@@ -1313,7 +1347,7 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
blockRoot = pblockindex->hashFinalSaplingRoot; blockRoot = pblockindex->hashFinalSaplingRoot;
if (witnessRoot == blockRoot) { if (witnessRoot == blockRoot) {
nd->witnessRootValidated = true; nd->witnessRootValidated = true;
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight); nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
continue; continue;
} }
} }
@@ -1365,10 +1399,7 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
} }
nd->witnessHeight = pblockindex->GetHeight(); nd->witnessHeight = pblockindex->GetHeight();
UpdateSaplingNullifierNoteMapWithTx(wtxItem.second); UpdateSaplingNullifierNoteMapWithTx(wtxItem.second);
if (item.second.nullifier) nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
else
LogPrintf("%s: no witness found for note %s:%d in its block; it stays unspendable until a rescan\n", __func__, wtxHash.ToString(), op.n);
} }
} }
} }
@@ -1487,17 +1518,25 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
work.reserve(pending.size()); work.reserve(pending.size());
for (const PendingNote& p : pending) work.push_back(p.nd); for (const PendingNote& p : pending) work.push_back(p.nd);
// Only a substantial bulk rebuild (e.g. a one-time post-upgrade repair) is worth parallelizing // Only a substantial bulk rebuild (e.g. a one-time post-upgrade repair) is logged and has its
// and logging; routine 1-block tip extension runs serially to avoid per-block thread-spawn // anchors checked; routine 1-block tip extension stays quiet to avoid per-block log spam.
// overhead and log spam.
const bool bulkRebuild = (rangeLen > 100); const bool bulkRebuild = (rangeLen > 100);
// Parallelize whenever the extension itself is expensive, not only for a bulk range: a routine
// 1-block extension appends each of the block's commitments to every lagging note, so a large
// wallet and a busy block (13k notes x 20 commitments) is seconds of hashing under
// cs_main/cs_wallet. Spawning the threads costs ~1ms, so only small jobs stay serial.
uint64_t nCms = 0;
for (const std::vector<uint256>& cms : blockCms)
nCms += cms.size();
const bool fParallel = bulkRebuild || (uint64_t)work.size() * nCms >= 2000;
int nPar = (int)GetArg("-witnessbuildthreads", 0); int nPar = (int)GetArg("-witnessbuildthreads", 0);
if (nPar <= 0) nPar = (int)std::thread::hardware_concurrency(); if (nPar <= 0) nPar = (int)std::thread::hardware_concurrency();
if (nPar <= 0) nPar = 1; if (nPar <= 0) nPar = 1;
if (nPar > (int)work.size()) nPar = (int)work.size(); if (nPar > (int)work.size()) nPar = (int)work.size();
if (nPar < 1) nPar = 1; if (nPar < 1) nPar = 1;
if (!bulkRebuild) nPar = 1; if (!fParallel) nPar = 1;
std::atomic<bool> failed(false); std::atomic<bool> failed(false);
std::mutex failMtx; std::mutex failMtx;
@@ -1509,14 +1548,27 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
// clone-every-block path for A/B verification. // clone-every-block path for A/B verification.
bool fastRebuild = GetBoolArg("-witnessfastrebuild", true); bool fastRebuild = GetBoolArg("-witnessfastrebuild", true);
// A bulk rebuild reports how many notes failed to reach the canonical anchor. root() walks the
// full depth-32 path (~2.4ms), so the check runs here on the workers and never on the routine
// per-block path, where it used to hold cs_main/cs_wallet for ~30s per block on a ~13k-note
// wallet purely to feed a log line that path never prints.
const uint256 canonicalRoot = pindex->hashFinalSaplingRoot;
std::atomic<int> nUnrecoverable(0);
auto checkRoot = [&](const SaplingNoteData* nd) {
if (bulkRebuild && (nd->witnesses.empty() || nd->witnesses.front().root() != canonicalRoot))
nUnrecoverable++;
};
auto worker = [&](int tid) { auto worker = [&](int tid) {
try { try {
for (size_t k = (size_t)tid; k < work.size(); k += (size_t)nPar) { for (size_t k = (size_t)tid; k < work.size(); k += (size_t)nPar) {
SaplingNoteData* nd = work[k]; SaplingNoteData* nd = work[k];
int startH = nd->witnessHeight; int startH = nd->witnessHeight;
int nBlocks = tipHeight - startH; int nBlocks = tipHeight - startH;
if (nBlocks <= 0) if (nBlocks <= 0) {
checkRoot(nd);
continue; continue;
}
if (!fastRebuild || nBlocks <= CACHE) { if (!fastRebuild || nBlocks <= CACHE) {
// Reference / shallow path: clone every block (preserves pre-existing older snapshots). // Reference / shallow path: clone every block (preserves pre-existing older snapshots).
@@ -1557,6 +1609,7 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
nd->witnesses.pop_back(); nd->witnesses.pop_back();
nd->witnessHeight = tipHeight; nd->witnessHeight = tipHeight;
} }
checkRoot(nd);
} }
} catch (const std::exception& e) { } catch (const std::exception& e) {
std::lock_guard<std::mutex> lk(failMtx); std::lock_guard<std::mutex> lk(failMtx);
@@ -1571,9 +1624,18 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
if (nPar <= 1) { if (nPar <= 1) {
worker(0); worker(0);
} else { } else {
// If the OS refuses a thread (task limit, memory), run the remaining shards here instead of
// letting the exception unwind past joinable threads (std::terminate). Each shard still runs
// exactly once, so the result is unchanged.
std::vector<std::thread> threads; std::vector<std::thread> threads;
threads.reserve(nPar); threads.reserve(nPar);
for (int t = 0; t < nPar; t++) threads.emplace_back(worker, t); int t = 0;
try {
for (; t < nPar; t++) threads.emplace_back(worker, t);
} catch (const std::system_error& e) {
LogPrintf("%s: could only start %d of %d witness threads (%s); running the rest inline\n", __func__, t, nPar, e.what());
}
for (int u = t; u < nPar; u++) worker(u);
for (std::thread& th : threads) th.join(); for (std::thread& th : threads) th.join();
} }
int64_t tExtend = GetTimeMicros() - e0; int64_t tExtend = GetTimeMicros() - e0;
@@ -1588,20 +1650,16 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
// is unrecoverable here: it is left flagged (the GetSaplingNoteWitnesses majority-anchor guard // is unrecoverable here: it is left flagged (the GetSaplingNoteWitnesses majority-anchor guard
// skips it for spends) and reported, rather than spun on indefinitely. witnessRootValidated is // skips it for spends) and reported, rather than spun on indefinitely. witnessRootValidated is
// in-memory only, so a fresh validation pass still runs on each restart and after any reorg // in-memory only, so a fresh validation pass still runs on each restart and after any reorg
// (DecrementNoteWitnesses clears it), keeping the heal self-correcting. // (DecrementNoteWitnesses clears it), keeping the heal self-correcting. The unrecoverable count
// itself comes from checkRoot() on the workers, and only for a bulk rebuild.
if (!work.empty()) { if (!work.empty()) {
const uint256& canonicalRoot = pindex->hashFinalSaplingRoot; for (SaplingNoteData* nd : work)
int nUnrecoverable = 0;
for (SaplingNoteData* nd : work) {
if (nd->witnesses.empty() || nd->witnesses.front().root() != canonicalRoot)
nUnrecoverable++;
nd->witnessRootValidated = true; nd->witnessRootValidated = true;
}
if (bulkRebuild) { if (bulkRebuild) {
LogPrintf("%s: rebuilt %u note witness cache(s) to height %d in %ldms using %d thread(s)%s\n", LogPrintf("%s: rebuilt %u note witness cache(s) to height %d in %ldms using %d thread(s)%s\n",
__func__, (unsigned)work.size(), tipHeight, (long)((tRead + tExtend) / 1000), nPar, __func__, (unsigned)work.size(), tipHeight, (long)((tRead + tExtend) / 1000), nPar,
nUnrecoverable nUnrecoverable.load()
? strprintf(" [WARNING: %d note(s) could not be rebuilt to the canonical anchor and were skipped]", nUnrecoverable).c_str() ? strprintf(" [WARNING: %d note(s) could not be rebuilt to the canonical anchor and were skipped]", nUnrecoverable.load()).c_str()
: ""); : "");
} }
} }
@@ -1711,12 +1769,10 @@ int64_t CWallet::IncOrderPosNext(CWalletDB *pwalletdb)
return nRet; return nRet;
} }
CWallet::TxItems CWallet::OrderedTxItems(std::list<CAccountingEntry>& acentries, std::string strAccount, bool fReadOnlyDb) CWallet::TxItems CWallet::OrderedTxItems(std::list<CAccountingEntry>& acentries, std::string strAccount)
{ {
AssertLockHeld(cs_wallet); // mapWallet AssertLockHeld(cs_wallet); // mapWallet
// Only a read cursor over the accounting entries runs on this handle. Pure readers open it CWalletDB walletdb(strWalletFile);
// read-only and without flush-on-close, so the call does not end in a txn_checkpoint.
CWalletDB walletdb(strWalletFile, fReadOnlyDb ? "r" : "r+", !fReadOnlyDb);
// First: get all CWalletTx and CAccountingEntry into a sorted-by-order multimap. // First: get all CWalletTx and CAccountingEntry into a sorted-by-order multimap.
TxItems txOrdered; TxItems txOrdered;
@@ -1925,6 +1981,8 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
bool fUpdated = false; bool fUpdated = false;
if (!fInsertedNew) if (!fInsertedNew)
{ {
// Re-mined in a different block (a reorg): its notes sit at new tree positions.
bool fSameBlock = wtx.hashBlock.IsNull() || wtxIn.hashBlock.IsNull() || wtxIn.hashBlock == wtx.hashBlock;
// Merge // Merge
if (!wtxIn.hashBlock.IsNull() && wtxIn.hashBlock != wtx.hashBlock) if (!wtxIn.hashBlock.IsNull() && wtxIn.hashBlock != wtx.hashBlock)
{ {
@@ -1937,7 +1995,7 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
wtx.nIndex = wtxIn.nIndex; wtx.nIndex = wtxIn.nIndex;
fUpdated = true; fUpdated = true;
} }
if (UpdatedNoteData(wtxIn, wtx)) { if (UpdatedNoteData(wtxIn, wtx, fSameBlock)) {
fUpdated = true; fUpdated = true;
} }
if (wtxIn.fFromMe && wtxIn.fFromMe != wtx.fFromMe) if (wtxIn.fFromMe && wtxIn.fFromMe != wtx.fFromMe)
@@ -1974,29 +2032,92 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
return true; return true;
} }
bool CWallet::UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx) // True if both maps hold the same notes with identical persisted state (every field SaplingNoteData
// serializes). SaplingNoteData::operator== ignores the witnesses, so it cannot tell a real change
// from a redundant update.
// Copies the decrypted plaintext of each incoming note into the matching (same outpoint and ivk) entry that
// lacks one. UpdatedNoteData uses it when it keeps the wallet's entries: a replay or rescan has just
// decrypted the notes (FindMySaplingNotes), and entries loaded from wallet.dat have no plaintext. The cache
// is in memory only and outside every comparison, so this never makes a tx look changed.
static void KeepNotePlaintexts(const mapSaplingNoteData_t& incoming, mapSaplingNoteData_t& wallet)
{
for (const mapSaplingNoteData_t::value_type& in : incoming) {
if (!in.second.cachedPlaintext)
continue;
mapSaplingNoteData_t::iterator it = wallet.find(in.first);
if (it != wallet.end() && !it->second.cachedPlaintext && it->second.ivk == in.second.ivk)
it->second.cachedPlaintext = in.second.cachedPlaintext;
}
}
static bool SaplingNoteDataIdentical(const mapSaplingNoteData_t& a, const mapSaplingNoteData_t& b)
{
if (a.size() != b.size())
return false;
for (const mapSaplingNoteData_t::value_type& item : a) {
mapSaplingNoteData_t::const_iterator it = b.find(item.first);
if (it == b.end())
return false;
const SaplingNoteData& x = item.second;
const SaplingNoteData& y = it->second;
if (!(x.ivk == y.ivk) || x.nullifier != y.nullifier || x.witnessHeight != y.witnessHeight || x.witnesses != y.witnesses)
return false;
}
return true;
}
bool CWallet::UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock)
{ {
bool unchangedSaplingFlag = (wtxIn.mapSaplingNoteData.empty() || wtxIn.mapSaplingNoteData == wtx.mapSaplingNoteData); bool unchangedSaplingFlag = (wtxIn.mapSaplingNoteData.empty() || wtxIn.mapSaplingNoteData == wtx.mapSaplingNoteData);
if (!unchangedSaplingFlag) { if (unchangedSaplingFlag) {
auto tmp = wtxIn.mapSaplingNoteData; KeepNotePlaintexts(wtxIn.mapSaplingNoteData, wtx.mapSaplingNoteData);
// Ensure we keep any cached witnesses we may already have return false;
for (const std::pair <SaplingOutPoint, SaplingNoteData> nd : wtx.mapSaplingNoteData) {
if (tmp.count(nd.first) && nd.second.witnesses.size() > 0) {
tmp.at(nd.first).witnesses.assign(
nd.second.witnesses.cbegin(), nd.second.witnesses.cend());
//PART B2: only carry over witnessHeight TOGETHER with the witnesses it describes.
//Copying it unconditionally (when witnesses are NOT copied) advances the height past
//the actual witness state for a whole tx's notes at once = the batch desync originator.
tmp.at(nd.first).witnessHeight = nd.second.witnessHeight;
}
}
// Now copy over the updated note data
wtx.mapSaplingNoteData = tmp;
} }
return !unchangedSaplingFlag; auto tmp = wtxIn.mapSaplingNoteData;
// Ensure we keep any cached witnesses we may already have
for (const mapSaplingNoteData_t::value_type& nd : wtx.mapSaplingNoteData) {
if (tmp.count(nd.first) && nd.second.witnesses.size() > 0) {
tmp.at(nd.first).witnesses.assign(
nd.second.witnesses.cbegin(), nd.second.witnesses.cend());
//PART B2: only carry over witnessHeight TOGETHER with the witnesses it describes.
//Copying it unconditionally (when witnesses are NOT copied) advances the height past
//the actual witness state for a whole tx's notes at once = the batch desync originator.
tmp.at(nd.first).witnessHeight = nd.second.witnessHeight;
//The nullifier is derived from the note and its witness position, so it travels with the
//witnesses too. wtxIn comes from FindMySaplingNotes, which never sets it; dropping it made
//VerifyAndSetInitialWitness clear and reseed the witnesses just carried over, so replaying
//or rescanning already-witnessed txs forced a rebuild of every one of their notes. A tx
//re-mined in another block is the exception: its notes moved, so the old nullifier is
//wrong and dropping it still reseeds them at the new position.
if (nd.second.nullifier && fSameBlock)
tmp.at(nd.first).nullifier = nd.second.nullifier;
}
}
// Keep a decrypted plaintext the wallet already holds (in memory only; a pure function of the
// outpoint and ivk) when the incoming note data lacks one.
for (const mapSaplingNoteData_t::value_type& nd : wtx.mapSaplingNoteData) {
mapSaplingNoteData_t::iterator it = tmp.find(nd.first);
if (it != tmp.end() && !it->second.cachedPlaintext && nd.second.cachedPlaintext && it->second.ivk == nd.second.ivk)
it->second.cachedPlaintext = nd.second.cachedPlaintext;
}
// A replayed or rescanned tx usually carries nothing new; don't rewrite it to disk. Do have
// VerifyAndSetInitialWitness re-check the kept witnesses against the active chain once (the
// flag is in memory only): the chain may have moved under them, and it keeps a rescan able to
// find and reseed a witness that went bad, without rebuilding the healthy ones.
if (SaplingNoteDataIdentical(tmp, wtx.mapSaplingNoteData)) {
for (mapSaplingNoteData_t::value_type& item : wtx.mapSaplingNoteData)
item.second.witnessRootValidated = false;
KeepNotePlaintexts(tmp, wtx.mapSaplingNoteData);
return false;
}
// Now copy over the updated note data
wtx.mapSaplingNoteData = tmp;
return true;
} }
/** /**
@@ -2169,21 +2290,24 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
// Protocol Spec: 4.19 Block Chain Scanning (Sapling) // Protocol Spec: 4.19 Block Chain Scanning (Sapling)
for (uint32_t i = 0; i < nZouts; ++i) { for (uint32_t i = 0; i < nZouts; ++i) {
const OutputDescription output = tx.vShieldedOutput[i]; const OutputDescription& output = tx.vShieldedOutput[i];
bool found = false; bool found = false;
for (auto it = mapSaplingFullViewingKeys.begin(); it != mapSaplingFullViewingKeys.end(); ++it) { for (auto it = mapSaplingFullViewingKeys.begin(); it != mapSaplingFullViewingKeys.end(); ++it) {
SaplingIncomingViewingKey ivk = it->first; SaplingIncomingViewingKey ivk = it->first;
auto result = SaplingNotePlaintext::decrypt(output.encCiphertext, ivk, output.ephemeralKey, output.cm); auto result = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
if (result) { if (result) {
auto address = ivk.address(result.get().d); // The note's (d, pk_d) is the address it was sent to, exactly what ivk.address(d) would
if (address && mapSaplingIncomingViewingKeys.count(address.get()) == 0) { // derive again: decrypt_with_note already computed pk_d from this ivk and d.
viewingKeysToAdd[address.get()] = ivk; const libzcash::SaplingPaymentAddress address(result->second.d, result->second.pk_d);
if (mapSaplingIncomingViewingKeys.count(address) == 0) {
viewingKeysToAdd[address] = ivk;
} }
// We don't cache the nullifier here as computing it requires knowledge of the note position // We don't cache the nullifier here as computing it requires knowledge of the note position
// in the commitment tree, which can only be determined when the transaction has been mined. // in the commitment tree, which can only be determined when the transaction has been mined.
SaplingOutPoint op {hash, i}; SaplingOutPoint op {hash, i};
SaplingNoteData nd; SaplingNoteData nd;
nd.ivk = ivk; nd.ivk = ivk;
nd.cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
noteData.insert(std::make_pair(op, nd)); noteData.insert(std::make_pair(op, nd));
found = true; found = true;
break; break;
@@ -2192,7 +2316,7 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
if (!found) { if (!found) {
for (auto it = mapSaplingIncomingViewingKeys.begin(); it != mapSaplingIncomingViewingKeys.end(); ++it) { for (auto it = mapSaplingIncomingViewingKeys.begin(); it != mapSaplingIncomingViewingKeys.end(); ++it) {
SaplingIncomingViewingKey ivk = it-> second; SaplingIncomingViewingKey ivk = it-> second;
auto result = SaplingNotePlaintext::decrypt(output.encCiphertext, ivk, output.ephemeralKey, output.cm); auto result = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
if (!result) { if (!result) {
continue; continue;
} }
@@ -2201,6 +2325,7 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
SaplingOutPoint op {hash, i}; SaplingOutPoint op {hash, i};
SaplingNoteData nd; SaplingNoteData nd;
nd.ivk = ivk; nd.ivk = ivk;
nd.cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
noteData.insert(std::make_pair(op, nd)); noteData.insert(std::make_pair(op, nd));
break; break;
} }
@@ -2947,13 +3072,12 @@ boost::optional<std::pair<
SaplingPaymentAddress>> CWalletTx::DecryptSaplingNote(SaplingOutPoint op) const SaplingPaymentAddress>> CWalletTx::DecryptSaplingNote(SaplingOutPoint op) const
{ {
// Check whether we can decrypt this SaplingOutPoint // Check whether we can decrypt this SaplingOutPoint
mapSaplingNoteData_t::const_iterator ndIt = this->mapSaplingNoteData.find(op); if (this->mapSaplingNoteData.count(op) == 0) {
if (ndIt == this->mapSaplingNoteData.end()) {
return boost::none; return boost::none;
} }
const OutputDescription& output = this->vShieldedOutput[op.n]; auto output = this->vShieldedOutput[op.n];
const SaplingNoteData& nd = ndIt->second; auto nd = this->mapSaplingNoteData.at(op);
auto maybe_pt = SaplingNotePlaintext::decrypt( auto maybe_pt = SaplingNotePlaintext::decrypt(
output.encCiphertext, output.encCiphertext,
@@ -5622,6 +5746,137 @@ bool CMerkleTx::AcceptToMemoryPool(bool fLimitFree, bool fRejectAbsurdFee)
return ::AcceptToMemoryPool(mempool, state, *this, fLimitFree, NULL, fRejectAbsurdFee); return ::AcceptToMemoryPool(mempool, state, *this, fLimitFree, NULL, fRejectAbsurdFee);
} }
unsigned int CWallet::FillSaplingNoteCache(const std::vector<SaplingNoteToDecrypt>& notes)
{
AssertLockHeld(cs_wallet);
// Each decryption is independent: it reads its own output and writes only its own note's
// cachedPlaintext. So a big batch (a whole wallet after a restart, ~0.5ms a note) is split across
// threads the way BuildWitnessCache's Phase 2 is, while this thread keeps cs_wallet held. Workers
// must not lock, or even assert a lock: under DEBUG_LOCKORDER a fresh thread has no lock stack.
// Spawning costs ~1ms, so small batches stay serial.
int nPar = 1;
if (notes.size() >= 128) {
nPar = (int)GetArg("-witnessbuildthreads", 0);
if (nPar <= 0) nPar = (int)std::thread::hardware_concurrency();
nPar = std::max(1, std::min(nPar, (int)(notes.size() / 64)));
}
std::vector<unsigned int> filled(nPar, 0);
auto worker = [&](int tid) {
for (size_t k = (size_t)tid; k < notes.size(); k += (size_t)nPar) {
const SaplingNoteToDecrypt& n = notes[k];
try {
auto result = SaplingNotePlaintext::decrypt_with_note(
n.output->encCiphertext, n.nd->ivk, n.output->ephemeralKey, n.output->cm);
if (result) {
n.nd->cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
filled[tid]++;
}
} catch (...) {
// Left unfilled, exactly like a note that doesn't decrypt; callers handle that. Nothing may
// escape a std::thread.
}
}
};
if (nPar <= 1) {
worker(0);
} else {
// If the OS refuses a thread, run the remaining shards here instead of letting the exception
// unwind past joinable threads (std::terminate). Each shard still runs exactly once.
std::vector<std::thread> threads;
threads.reserve(nPar);
int t = 0;
try {
for (; t < nPar; t++) threads.emplace_back(worker, t);
} catch (const std::exception& e) {
LogPrintf("%s: could only start %d of %d threads (%s); decrypting the rest inline\n", __func__, t, nPar, e.what());
}
for (int u = t; u < nPar; u++) worker(u);
for (std::thread& th : threads) th.join();
}
unsigned int nFilled = 0;
for (unsigned int f : filled)
nFilled += f;
nSaplingNoteCacheFills += nFilled;
nSaplingNoteCacheFillThreads = nPar;
LogPrint("bench", "%s: decrypted %u of %u notes using %d thread(s)\n", __func__, nFilled, (unsigned int)notes.size(), nPar);
return nFilled;
}
// wallet.dat doesn't store the plaintext, so without this the first balance call after a restart decrypts
// the whole wallet under cs_main (~0.5ms a note). init runs it before RPC leaves warmup and before the
// wallet notification thread starts; only the few RPCs allowed during warmup (key export, address
// listing) can wait on cs_wallet meanwhile. Unlike GetFilteredNotes it never asserts, so a corrupt note
// can't stop the node from starting (or from serving those RPCs); GetFilteredNotes still asserts on such
// a note, as before.
CWallet::SaplingNoteCacheWarmup CWallet::WarmSaplingNoteCache()
{
const std::chrono::steady_clock::time_point start = std::chrono::steady_clock::now();
LOCK(cs_wallet);
SaplingNoteCacheWarmup result;
std::vector<SaplingNoteToDecrypt> uncached;
for (auto& p : mapWallet) {
CWalletTx& wtx = p.second;
for (auto& item : wtx.mapSaplingNoteData) {
result.notes++;
if (item.second.cachedPlaintext)
continue;
if (item.first.n >= wtx.vShieldedOutput.size()) {
LogPrint("zrpcunsafe", "%s: note %s:%u has no such output\n", __func__, item.first.hash.ToString(), item.first.n);
result.noSuchOutput++;
continue;
}
uncached.push_back(SaplingNoteToDecrypt { &item.second, &wtx.vShieldedOutput[item.first.n] });
}
}
result.filled = uncached.empty() ? 0 : FillSaplingNoteCache(uncached);
for (const SaplingNoteToDecrypt& n : uncached) {
if (!n.nd->cachedPlaintext) {
LogPrint("zrpcunsafe", "%s: a note of output %s does not decrypt under its ivk\n", __func__, n.output->cm.ToString());
result.undecryptable++;
}
}
const unsigned int nBad = result.noSuchOutput + result.undecryptable;
const int64_t nMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - start).count();
LogPrintf("%s: decrypted %u of %u Sapling notes in %dms%s\n", __func__, result.filled, result.notes, nMs,
nBad ? strprintf("; %u could not be decrypted (-debug=zrpcunsafe lists them)", nBad) : "");
return result;
}
// -checknotecache: decrypts a cached note again, and logs and repairs any difference. GetFilteredNotes runs
// it on every note that passes its tx, spent and locked filters, before the address filter, so a corrupt
// cached address can't wrongly drop a note unchecked. A diagnostic for the cache: it costs the full
// decryption the cache exists to avoid.
static void CheckSaplingNoteCache(const SaplingOutPoint& op, SaplingNoteData& nd, const OutputDescription& output)
{
auto fresh = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, nd.ivk, output.ephemeralKey, output.cm);
if (!fresh) {
LogPrintf("%s: note %s:%u has a cached plaintext but no longer decrypts\n", __func__, op.hash.ToString(), op.n);
return;
}
const SaplingNoteCache& cached = *nd.cachedPlaintext;
const libzcash::SaplingNote& note = fresh->second;
if (!(cached.note.d == note.d && cached.note.pk_d == note.pk_d && cached.note.r == note.r &&
cached.note.value() == note.value() && cached.memo == fresh->first.memo())) {
LogPrintf("%s: cached plaintext of note %s:%u differs from a fresh decryption; using the fresh one\n",
__func__, op.hash.ToString(), op.n);
nd.cachedPlaintext = SaplingNoteCache{ note, fresh->first.memo() };
}
}
// Replaces tx_height() on the wallet's note paths. tx_height() re-read the tx from disk through
// GetTransaction, once per wallet tx (or per returned note) under LOCK2, and on a -txindex=0 node it
// never found a fully shielded tx, so those notes were silently skipped. The depth already locates
// the block: GetDepthInMainChain is chainActive.Height() - height + 1 for an active-chain tx.
int HeightFromDepth(int nDepth)
{
AssertLockHeld(cs_main);
return nDepth > 0 ? chainActive.Height() - nDepth + 1 : 0;
}
/** /**
* Find notes in the wallet filtered by payment address, min depth and ability to spend. * Find notes in the wallet filtered by payment address, min depth and ability to spend.
* These notes are decrypted and added to the output parameter vector, outEntries. * These notes are decrypted and added to the output parameter vector, outEntries.
@@ -5642,26 +5897,6 @@ void CWallet::GetFilteredNotes(
GetFilteredNotes(saplingEntries, filterAddresses, minDepth, INT_MAX, ignoreSpent, requireSpendingKey); GetFilteredNotes(saplingEntries, filterAddresses, minDepth, INT_MAX, ignoreSpent, requireSpendingKey);
} }
namespace {
// GetFilteredNotes' decrypt-free address pre-filter is only built for at most this many
// filter addresses. Verifying each one costs a Jubjub scalar multiplication, which buys
// nothing when the filter is "every address in the wallet" (z_listunspent, z_getbalances).
// Setting it to 0 disables the pre-filter entirely.
const size_t GETFILTEREDNOTES_PREFILTER_MAX_ADDRESSES = 16;
// True if ivk < 2^251. A genuine Sapling ivk is CRH_ivk output with its top 5 bits cleared
// (librustzcash_crh_ivk), so it is below 2^251 < r_J (the Jubjub subgroup order). Below r_J,
// pk_d = [ivk] g_d determines ivk uniquely. librustzcash_ivk_to_pkd multiplies by the raw
// 256-bit value without reducing it, so a value >= r_J aliases ivk - r_J. Byte 31 of the
// little-endian representation is the most significant byte.
bool IsCanonicalSaplingIvk(const libzcash::SaplingIncomingViewingKey& ivk)
{
return (ivk.begin()[31] & 0xF8) == 0;
}
} // namespace
/** /**
* Find notes in the wallet filtered by payment addresses, min depth, max depth, * Find notes in the wallet filtered by payment addresses, min depth, max depth,
* if the note is spent, if a spending key is required, and if the notes are locked. * if the note is spent, if a spending key is required, and if the notes are locked.
@@ -5678,47 +5913,28 @@ void CWallet::GetFilteredNotes(
{ {
LOCK2(cs_main, cs_wallet); LOCK2(cs_main, cs_wallet);
LogPrintf("%s ignoreLocked=%d\n", __func__, ignoreLocked); // Every caller already holds LOCK2, so the lock above never waits and this times the walk alone.
// Time spent waiting for the locks shows up in the RPC's own "took" line (rpc/server.cpp).
const std::chrono::steady_clock::time_point walkStart = std::chrono::steady_clock::now();
const size_t nEntriesBefore = saplingEntries.size();
unsigned int nTxs = 0, nNotes = 0, nSpent = 0;
// Decrypt-free address pre-filter. A note matches a Sapling filter address a only if // Pass 1: the notes that pass every filter not needing the plaintext, in wallet order.
// nd.ivk.address(a.d) == a. If a's recorded ivk r is canonical and provably generates a struct Candidate {
// (r.address(a.d) == a), then any other canonical ivk cannot generate a, so a note whose const SaplingOutPoint* op;
// canonical nd.ivk is not one of these r can be skipped without decrypting it. Recorded SaplingNoteData* nd;
// pairs are not trusted as-is (z_importviewingkey stores any (ivk, zaddr) pair the user const OutputDescription* output;
// supplies), so each one is verified. If any Sapling filter address has no recorded ivk, int nDepth;
// a non-canonical one, or one that does not generate it, the pre-filter is off and every };
// note takes the old decrypt path. InvalidEncoding entries never equal a note's address. std::vector<Candidate> candidates;
std::set<libzcash::SaplingIncomingViewingKey> prefilterIvks; std::vector<SaplingNoteToDecrypt> uncached;
bool usePrefilter = !filterAddresses.empty() && for (auto & p : mapWallet) {
filterAddresses.size() <= GETFILTEREDNOTES_PREFILTER_MAX_ADDRESSES; // By reference: a copy deep-copies every note's witness list (up to WITNESS_CACHE_SIZE
if (usePrefilter) { // witnesses each), and it was made before any filter ran. Not const: a missing plaintext is
for (const libzcash::PaymentAddress& fa : filterAddresses) { // cached onto the wallet's own entry.
const libzcash::SaplingPaymentAddress* sa = boost::get<libzcash::SaplingPaymentAddress>(&fa); CWalletTx& wtx = p.second;
if (sa == nullptr) {
continue;
}
libzcash::SaplingIncomingViewingKey rivk;
if (!GetSaplingIncomingViewingKey(*sa, rivk) || !IsCanonicalSaplingIvk(rivk)) {
usePrefilter = false;
break;
}
auto derived = rivk.address(sa->d);
if (!derived || !(derived.get() == *sa)) {
usePrefilter = false;
break;
}
prefilterIvks.insert(rivk);
}
}
for (const auto & p : mapWallet) { // Most wallet txs (coinbase, sends from other wallets) carry no notes of ours.
// A reference, not a copy: copying a CWalletTx deep-copies every note's witness list.
// Nothing in this loop mutates mapWallet or any CWalletTx, and cs_main + cs_wallet are
// held for the whole function, so the reference stays valid.
const CWalletTx& wtx = p.second;
// A tx without Sapling note data can never yield an entry (the note loop below runs
// zero times), so skip it before the finality, maturity and depth work.
if (wtx.mapSaplingNoteData.empty()) if (wtx.mapSaplingNoteData.empty())
continue; continue;
@@ -5726,84 +5942,97 @@ void CWallet::GetFilteredNotes(
if (!CheckFinalTx(wtx) || wtx.GetBlocksToMaturity() > 0) if (!CheckFinalTx(wtx) || wtx.GetBlocksToMaturity() > 0)
continue; continue;
// cs_main is held, so the depth can't change during the walk.
const int nDepth = wtx.GetDepthInMainChain();
if (minDepth > 1) { if (minDepth > 1) {
int nHeight = tx_height(wtx.GetHash()); int nHeight = HeightFromDepth(nDepth);
if ( nHeight == 0 ) { if ( nHeight == 0 ) {
continue; continue;
} }
int nDepth = wtx.GetDepthInMainChain();
int dpowconfs = hush_dpowconfs(nHeight,nDepth); int dpowconfs = hush_dpowconfs(nHeight,nDepth);
if ( dpowconfs < minDepth || dpowconfs > maxDepth) { if ( dpowconfs < minDepth || dpowconfs > maxDepth) {
continue; continue;
} }
} else { } else {
if ( wtx.GetDepthInMainChain() < minDepth || if ( nDepth < minDepth || nDepth > maxDepth) {
wtx.GetDepthInMainChain() > maxDepth) {
continue; continue;
} }
} }
nTxs++;
for (const auto & pair : wtx.mapSaplingNoteData) { for (auto & pair : wtx.mapSaplingNoteData) {
const SaplingOutPoint& op = pair.first; const SaplingOutPoint& op = pair.first;
const SaplingNoteData& nd = pair.second; SaplingNoteData& nd = pair.second;
nNotes++;
// Address pre-filter (see prefilterIvks above): a canonical ivk that is not one of // The cheap filters run before any decryption, which costs ~0.5ms of Jubjub arithmetic
// the verified filter ivks cannot generate any filter address, so the // per note. Spent notes stay in the wallet (-deletetx is off by default).
// filterAddresses check after decryption would reject this note anyway.
if (usePrefilter && IsCanonicalSaplingIvk(nd.ivk) && prefilterIvks.count(nd.ivk) == 0) {
continue;
}
// Spent check before trial decryption: it reads only nd.nullifier and wallet
// state (not the plaintext or the address), so spent notes are never decrypted.
if (ignoreSpent && nd.nullifier && IsSaplingSpent(*nd.nullifier)) { if (ignoreSpent && nd.nullifier && IsSaplingSpent(*nd.nullifier)) {
nSpent++;
continue; continue;
} }
uint256 pk_d;
auto maybe_pt = SaplingNotePlaintext::decrypt_with_pk_d(
wtx.vShieldedOutput[op.n].encCiphertext,
nd.ivk,
wtx.vShieldedOutput[op.n].ephemeralKey,
wtx.vShieldedOutput[op.n].cm,
pk_d);
assert(static_cast<bool>(maybe_pt));
auto notePt = maybe_pt.get();
// Same value as nd.ivk.address(notePt.d).get(): decryption succeeded, so
// librustzcash_ivk_to_pkd(nd.ivk, d) returned true (d is a valid diversifier, which
// is all check_diversifier tests) and wrote pk_d, the value address() recomputes.
SaplingPaymentAddress pa(notePt.d, pk_d);
// skip notes which belong to a different payment address in the wallet
if (!(filterAddresses.empty() || filterAddresses.count(pa))) {
continue;
}
// skip notes which cannot be spent
if (requireSpendingKey) {
libzcash::SaplingIncomingViewingKey ivk;
libzcash::SaplingFullViewingKey fvk;
if (!(GetSaplingIncomingViewingKey(pa, ivk) &&
GetSaplingFullViewingKey(ivk, fvk) &&
HaveSaplingSpendingKey(fvk))) {
continue;
}
}
// skip locked notes // skip locked notes
if (ignoreLocked && IsLockedNote(op)) { if (ignoreLocked && IsLockedNote(op)) {
LogPrintf("%s: skipping locked note %s\n", __func__, op.hash.ToString().substr(0,10).c_str()); LogPrint("zrpcunsafe", "%s: skipping locked note %s\n", __func__, op.hash.ToString().substr(0,10).c_str());
continue; continue;
} }
// Same value as notePt.note(nd.ivk).get(), which is assert(op.n < wtx.vShieldedOutput.size());
// SaplingNote(d, nd.ivk.address(d)->pk_d, value, rcm): pa is nd.ivk.address(d). const OutputDescription* output = &wtx.vShieldedOutput[op.n];
libzcash::SaplingNote note(notePt.d, pa.pk_d, notePt.value(), notePt.rcm); candidates.push_back(Candidate { &op, &nd, output, nDepth });
saplingEntries.push_back(SaplingNoteEntry { if (!nd.cachedPlaintext)
op, pa, note, notePt.memo(), wtx.GetDepthInMainChain() }); uncached.push_back(SaplingNoteToDecrypt { &nd, output });
} }
} }
// Decrypt what the cache lacks: normally nothing, since FindMySaplingNotes fills new notes and the
// startup warm-up fills the ones loaded from wallet.dat.
const unsigned int nFilled = uncached.empty() ? 0 : FillSaplingNoteCache(uncached);
// Pass 2: the filters that need the note's address.
if (filterAddresses.empty())
saplingEntries.reserve(saplingEntries.size() + candidates.size());
for (const Candidate& c : candidates) {
// A wallet note that doesn't decrypt under its own ivk is corrupt; as before, that's fatal here.
assert(static_cast<bool>(c.nd->cachedPlaintext));
if (fCheckNoteCache)
CheckSaplingNoteCache(*c.op, *c.nd, *c.output);
const SaplingNoteCache& plaintext = *c.nd->cachedPlaintext;
const libzcash::SaplingPaymentAddress pa(plaintext.note.d, plaintext.note.pk_d);
// skip notes which belong to a different payment address in the wallet
if (!(filterAddresses.empty() || filterAddresses.count(pa))) {
continue;
}
// skip notes which cannot be spent
if (requireSpendingKey) {
libzcash::SaplingIncomingViewingKey ivk;
libzcash::SaplingFullViewingKey fvk;
if (!(GetSaplingIncomingViewingKey(pa, ivk) &&
GetSaplingFullViewingKey(ivk, fvk) &&
HaveSaplingSpendingKey(fvk))) {
continue;
}
}
saplingEntries.push_back(SaplingNoteEntry {
*c.op, pa, plaintext.note, plaintext.memo, c.nDepth });
}
// A slow walk holds cs_main long enough to stall block connection, so it is always logged; a
// fast one only with -debug=bench (the balance RPCs run it on every GUI poll). The wallet-size
// counts go to the default log only with -debug=bench too.
const int64_t nWalkMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - walkStart).count();
if (LogAcceptCategory("bench")) {
LogPrintf("%s: minDepth=%d ignoreSpent=%d requireSpendingKey=%d ignoreLocked=%d txs=%u/%u notes=%u spent=%u decrypted=%u entries=%u walk=%dms\n",
__func__, minDepth, ignoreSpent, requireSpendingKey, ignoreLocked, nTxs, (unsigned int)mapWallet.size(),
nNotes, nSpent, nFilled, (unsigned int)(saplingEntries.size() - nEntriesBefore), nWalkMs);
} else if (nWalkMs >= 500) {
LogPrintf("%s: minDepth=%d ignoreSpent=%d requireSpendingKey=%d ignoreLocked=%d walk=%dms\n",
__func__, minDepth, ignoreSpent, requireSpendingKey, ignoreLocked, nWalkMs);
}
} }
src/wallet/wallet.h
+56 -5
View File
@@ -62,6 +62,8 @@ extern bool bSpendZeroConfChange;
extern bool fSendFreeTransactions; extern bool fSendFreeTransactions;
extern bool fPayAtLeastCustomFee; extern bool fPayAtLeastCustomFee;
extern bool fTxDeleteEnabled; extern bool fTxDeleteEnabled;
//! -checknotecache: re-decrypt cached Sapling notes in GetFilteredNotes and repair any difference
extern bool fCheckNoteCache;
extern bool fTxConflictDeleteEnabled; extern bool fTxConflictDeleteEnabled;
extern int fDeleteInterval; extern int fDeleteInterval;
extern unsigned int fDeleteTransactionsAfterNBlocks; extern unsigned int fDeleteTransactionsAfterNBlocks;
@@ -301,6 +303,14 @@ class SproutNoteData
}; };
/** A Sapling note's decrypted plaintext as the wallet uses it: the note (value, rcm, and the address it
* was sent to as (d, pk_d)) and its memo. */
struct SaplingNoteCache
{
libzcash::SaplingNote note;
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
};
class SaplingNoteData class SaplingNoteData
{ {
public: public:
@@ -328,6 +338,15 @@ public:
// read true and short-circuit the witness self-heal in VerifyAndSetInitialWitness. // read true and short-circuit the witness self-heal in VerifyAndSetInitialWitness.
bool witnessRootValidated = false; bool witnessRootValidated = false;
//In Memory Only
// The note's decrypted plaintext, so the balance and note-listing calls don't redo the Sapling
// decryption (several Jubjub scalar multiplications) for every note on every call. It is a pure
// function of (txid, output index, ivk), so nothing invalidates it. Never serialized, and left out
// of operator== and SaplingNoteDataIdentical on purpose: a filled and an unfilled entry persist the
// same state, so filling one must never make a tx look changed and get rewritten. Filled by
// FindMySaplingNotes when the note is found, otherwise by FillSaplingNoteCache.
boost::optional<SaplingNoteCache> cachedPlaintext;
ADD_SERIALIZE_METHODS; ADD_SERIALIZE_METHODS;
template <typename Stream, typename Operation> template <typename Stream, typename Operation>
@@ -366,6 +385,11 @@ struct SaplingNoteEntry
int confirmations; int confirmations;
}; };
/** Height of the active-chain block holding a wallet tx that is nDepth deep (as GetDepthInMainChain
* returns it, and SaplingNoteEntry::confirmations holds it), or 0 when the tx is unconfirmed or not
* in the active chain. Caller holds cs_main. */
int HeightFromDepth(int nDepth);
/** A transaction with a merkle branch linking it to the block chain. */ /** A transaction with a merkle branch linking it to the block chain. */
class CMerkleTx : public CTransaction class CMerkleTx : public CTransaction
{ {
@@ -895,7 +919,7 @@ public:
protected: protected:
int SaplingWitnessMinimumHeight(const uint256& nullifier, int nWitnessHeight, int nMinimumHeight); int SaplingWitnessMinimumHeight(const boost::optional<uint256>& nullifier, int nWitnessHeight, int nMinimumHeight);
/** /**
* pindex is the new tip being connected. * pindex is the new tip being connected.
@@ -965,8 +989,12 @@ private:
template <class T> template <class T>
void SyncMetaData(std::pair<typename TxSpendMap<T>::iterator, typename TxSpendMap<T>::iterator>); void SyncMetaData(std::pair<typename TxSpendMap<T>::iterator, typename TxSpendMap<T>::iterator>);
// Last block ThreadNotifyWallets delivered to ChainTip, i.e. the block the wallet's state
// reflects (guarded by cs_wallet). SetBestChain stamps this instead of the chain tip.
const CBlockIndex* pindexWalletTip = nullptr;
protected: protected:
bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx); bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock);
void MarkAffectedTransactionsDirty(const CTransaction& tx); void MarkAffectedTransactionsDirty(const CTransaction& tx);
/* the hd chain data model (chain counters) */ /* the hd chain data model (chain counters) */
@@ -1210,12 +1238,10 @@ public:
/** /**
* Get the wallet's activity log * Get the wallet's activity log
* @param fReadOnlyDb open wallet.dat read-only and without flush-on-close (pure readers such as
* listtransactions: no txn_checkpoint per call); false = read-write, flush-on-close
* @return multimap of ordered transactions and accounting entries * @return multimap of ordered transactions and accounting entries
* @warning Returned pointers are *only* valid within the scope of passed acentries * @warning Returned pointers are *only* valid within the scope of passed acentries
*/ */
TxItems OrderedTxItems(std::list<CAccountingEntry>& acentries, std::string strAccount = "", bool fReadOnlyDb = false); TxItems OrderedTxItems(std::list<CAccountingEntry>& acentries, std::string strAccount = "");
void MarkDirty(); void MarkDirty();
bool UpdateNullifierNoteMap(); bool UpdateNullifierNoteMap();
@@ -1296,6 +1322,8 @@ public:
bool CommitAutomatedTx(const CTransaction& tx); bool CommitAutomatedTx(const CTransaction& tx);
/** Saves witness caches and best block locator to disk. Overrides CValidationInterface. */ /** Saves witness caches and best block locator to disk. Overrides CValidationInterface. */
void SetBestChain(const CBlockLocator& loc); void SetBestChain(const CBlockLocator& loc);
// Seed the block the wallet's state reflects, before ThreadNotifyWallets starts delivering.
void SetWalletTip(const CBlockIndex* pindex);
/** As SetBestChain, but for use INSIDE a long scan: opens the wallet DB with fFlushOnClose=false /** As SetBestChain, but for use INSIDE a long scan: opens the wallet DB with fFlushOnClose=false
* so the call does not trigger a full BDB txn_checkpoint over the whole cache (see the comment * so the call does not trigger a full BDB txn_checkpoint over the whole cache (see the comment
* at CWallet::SetBestChain), and reports whether the write actually committed. */ * at CWallet::SetBestChain), and reports whether the write actually committed. */
@@ -1460,6 +1488,29 @@ public:
bool LoadCryptedMnemonicEntropy(const uint256& entropyFp, const std::vector<unsigned char>& vchCryptedSecret); bool LoadCryptedMnemonicEntropy(const uint256& entropyFp, const std::vector<unsigned char>& vchCryptedSecret);
/* Find notes filtered by payment address, min depth, ability to spend */ /* Find notes filtered by payment address, min depth, ability to spend */
/** A wallet note whose decrypted plaintext is not cached yet, with the output it decrypts. */
struct SaplingNoteToDecrypt {
SaplingNoteData* nd;
const OutputDescription* output;
};
/** Decrypts each note into nd->cachedPlaintext and returns how many it filled. A note that doesn't
* decrypt under its ivk is left unfilled. Caller holds cs_wallet. */
unsigned int FillSaplingNoteCache(const std::vector<SaplingNoteToDecrypt>& notes);
/** Notes FillSaplingNoteCache has decrypted since startup (FindMySaplingNotes' fills not counted), and
* the threads its last call used. Guarded by cs_wallet. */
uint64_t nSaplingNoteCacheFills = 0;
int nSaplingNoteCacheFillThreads = 0;
struct SaplingNoteCacheWarmup {
unsigned int notes = 0; //!< notes in the wallet
unsigned int filled = 0; //!< decrypted by this call
unsigned int noSuchOutput = 0; //!< naming an output their tx doesn't have
unsigned int undecryptable = 0; //!< not decrypting under their own ivk
};
/** Decrypts every wallet note whose plaintext isn't cached yet (all of them after wallet.dat is loaded).
* Never asserts: a note that can't be decrypted is counted, logged and left unfilled. Run at startup. */
SaplingNoteCacheWarmup WarmSaplingNoteCache();
void GetFilteredNotes(std::vector<SaplingNoteEntry>& saplingEntries, void GetFilteredNotes(std::vector<SaplingNoteEntry>& saplingEntries,
std::string address, std::string address,
int minDepth=1, int minDepth=1,
src/zcash/Note.cpp
+35 -14
View File
@@ -115,23 +115,16 @@ boost::optional<SaplingOutgoingPlaintext> SaplingOutgoingPlaintext::decrypt(
} }
} }
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt( namespace {
const SaplingEncCiphertext &ciphertext,
const uint256 &ivk,
const uint256 &epk,
const uint256 &cmu
)
{
uint256 pk_d;
return decrypt_with_pk_d(ciphertext, ivk, epk, cmu, pk_d);
}
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt_with_pk_d( // SaplingNotePlaintext::decrypt with an incoming viewing key, also handing back the pk_d = [ivk]g_d
// it derives to check cmu.
boost::optional<SaplingNotePlaintext> DecryptWithIvk(
const SaplingEncCiphertext &ciphertext, const SaplingEncCiphertext &ciphertext,
const uint256 &ivk, const uint256 &ivk,
const uint256 &epk, const uint256 &epk,
const uint256 &cmu, const uint256 &cmu,
uint256 &pk_d_out uint256 &pk_d
) )
{ {
auto pt = AttemptSaplingEncDecryption(ciphertext, ivk, epk); auto pt = AttemptSaplingEncDecryption(ciphertext, ivk, epk);
@@ -152,7 +145,6 @@ boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt_with_pk_d(
return boost::none; return boost::none;
} }
uint256 pk_d;
if (!librustzcash_ivk_to_pkd(ivk.begin(), ret.d.data(), pk_d.begin())) { if (!librustzcash_ivk_to_pkd(ivk.begin(), ret.d.data(), pk_d.begin())) {
return boost::none; return boost::none;
} }
@@ -173,10 +165,39 @@ boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt_with_pk_d(
return boost::none; return boost::none;
} }
pk_d_out = pk_d;
return ret; return ret;
} }
}
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
const SaplingEncCiphertext &ciphertext,
const uint256 &ivk,
const uint256 &epk,
const uint256 &cmu
)
{
uint256 pk_d;
return DecryptWithIvk(ciphertext, ivk, epk, cmu, pk_d);
}
boost::optional<std::pair<SaplingNotePlaintext, SaplingNote>> SaplingNotePlaintext::decrypt_with_note(
const SaplingEncCiphertext &ciphertext,
const uint256 &ivk,
const uint256 &epk,
const uint256 &cmu
)
{
uint256 pk_d;
auto pt = DecryptWithIvk(ciphertext, ivk, epk, cmu, pk_d);
if (!pt) {
return boost::none;
}
// ivk_to_pkd succeeding implies d is a valid diversifier, which is all note(ivk) checks on top.
SaplingNote note(pt->d, pk_d, pt->value(), pt->rcm);
return std::make_pair(*pt, note);
}
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt( boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
const SaplingEncCiphertext &ciphertext, const SaplingEncCiphertext &ciphertext,
const uint256 &epk, const uint256 &epk,
src/zcash/Note.hpp
+11 -11
View File
@@ -77,17 +77,6 @@ public:
const uint256 &cmu const uint256 &cmu
); );
// Same as decrypt(ciphertext, ivk, epk, cmu). On success it also stores in pk_d_out the
// pk_d = [ivk] g_d that decryption computed, which equals ivk.address(d)->pk_d, so the
// caller does not have to recompute it. pk_d_out is not written on failure.
static boost::optional<SaplingNotePlaintext> decrypt_with_pk_d(
const SaplingEncCiphertext &ciphertext,
const uint256 &ivk,
const uint256 &epk,
const uint256 &cmu,
uint256 &pk_d_out
);
static boost::optional<SaplingNotePlaintext> decrypt( static boost::optional<SaplingNotePlaintext> decrypt(
const SaplingEncCiphertext &ciphertext, const SaplingEncCiphertext &ciphertext,
const uint256 &epk, const uint256 &epk,
@@ -96,6 +85,17 @@ public:
const uint256 &cmu const uint256 &cmu
); );
// decrypt(ciphertext, ivk, epk, cmu) that also returns the note. Building the note needs
// pk_d = [ivk]g_d, which the decryption derives anyway to check cmu; note(ivk) would derive it
// again (another Jubjub scalar multiplication). A distinct name rather than an overload, so it
// can never capture a call meant for the (epk, esk, pk_d, cmu) one.
static boost::optional<std::pair<SaplingNotePlaintext, SaplingNote>> decrypt_with_note(
const SaplingEncCiphertext &ciphertext,
const uint256 &ivk,
const uint256 &epk,
const uint256 &cmu
);
boost::optional<SaplingNote> note(const SaplingIncomingViewingKey& ivk) const; boost::optional<SaplingNote> note(const SaplingIncomingViewingKey& ivk) const;
virtual ~SaplingNotePlaintext() {} virtual ~SaplingNotePlaintext() {}