DragonX/dragonx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Compare commits

merge into: DragonX/dragonx:perf/wallet-scan-rpc-costs
Branches Tags
DragonX/dragonx:master DragonX/dragonx:dev DragonX/dragonx:perf/wallet-scan-rpc-costs DragonX/dragonx:audit-fixes-20260901 DragonX/dragonx:autoshield-safety DragonX/dragonx:feature/autoshield-coinbase
DragonX/dragonx:v1.3.0 DragonX/dragonx:v1.2.0 DragonX/dragonx:v1.1.0 DragonX/dragonx:v1.0.3 DragonX/dragonx:v1.0.2 DragonX/dragonx:v1.0.1 DragonX/dragonx:v1.0.0
...
pull from: DragonX/dragonx:dev
Branches Tags
DragonX/dragonx:dev DragonX/dragonx:perf/wallet-scan-rpc-costs DragonX/dragonx:master DragonX/dragonx:audit-fixes-20260901 DragonX/dragonx:autoshield-safety DragonX/dragonx:feature/autoshield-coinbase
DragonX/dragonx:v1.3.0 DragonX/dragonx:v1.2.0 DragonX/dragonx:v1.1.0 DragonX/dragonx:v1.0.3 DragonX/dragonx:v1.0.2 DragonX/dragonx:v1.0.1 DragonX/dragonx:v1.0.0
25 Commits
perf/wallet-scan-rpc-costs ... dev
Author SHA1 Message Date
DanSandClaude Opus 5.5 3032eb7868 wallet/rpc: gather Sapling keys once per history RPC, not per output per address 
zsWalletTxJSON (z_listsentbyaddress, z_listreceivedaddress, getalldata)
runs once per wallet tx under LOCK2(cs_main, cs_wallet). For every
shielded output it copied the wallet's address set, and for each address
fetched the spending key, re-derived its full viewing key (an
elliptic-curve operation), and trial-decrypted the output; the sends side
did the same with the ovk. Every received or sent note in a tx with
spends walked the whole wallet in GetNullifiersForAddresses for the
change check. The spends side decrypted every spent note again and
deep-copied its SaplingNoteData (witnesses included). The GUI runs
z_listsentbyaddress over the whole wallet on every launch.

SaplingHistoryKeys, built once per RPC call, holds each address with a
usable spending key (in GetSaplingPaymentAddresses order, with its ivk
and ovk as the spending key gives them, and its ivk as
GetSaplingIncomingViewingKey gives it). It also gives the change check
as a lookup into the wallet's (ivk, nullifier) pairs, collected on first
use. The helpers keep their exact structure and quirks. Each output's
JSON object and change flag still accumulate across the addresses that
decrypt it. Diversified addresses sharing a key still produce one entry
each. An unknown address still checks change under a zero ivk. A locked
wallet still reports no shielded received or sent entries. Only the
repeated work is gone:

- each output is decrypted at most once per distinct key. Once it is
  known which ivk decrypts an output, no other ivk is tried: the cached
  plaintext, or a decryption under the ivk the wallet's note data
  records. Two canonical ivks can't both pass the note commitment check,
  which pins pk_d = [ivk]g_d;
- outCiphertexts are decrypted once per distinct ovk; the t->z shielding
  ovk is derived from the seed once per tx;
- spends use the cached plaintext of the spent note.

Two deliberate behaviour changes. The spends helper looked notes up with
operator[]: it inserted a null-outpoint entry into
mapSaplingNullifiersToNotes for every other wallet's nullifier it met.
After getalldata or z_listsentbyaddress had run, z_viewtransaction of
any tx spending someone else's note (every incoming z->z payment) failed
with map::at. And with an inconsistent nullifier map, it inserted an
empty note into a wallet tx. The helpers now only read the wallet. Also,
if the walletpassphrase timer relocks the wallet during the call, the
whole call reports the keys as of its start; the old per-output lookups
switched over partway.

test_wallet_history.cpp holds 69bb035e7's helpers verbatim (renamed)
and a wallet exercising every branch:
- default and diversified addresses, a viewing-key-only address, and a
  key imported after its note arrived;
- z->z sends with change, and with text, empty and binary memos;
- a send nobody's ovk reads, and a t->z shielding send;
- our spends, someone else's, and a mixed tx.
Every tx x every return type x nine address filters gives byte-identical
JSON: with the plaintext cache warm, cold, and on an encrypted wallet
both locked and unlocked. Mutation-checked: dropping the ivk check from
the uniqueness shortcut fails the test. DoesNotModifyTheWallet pins the
read-only change (and shows that the old helpers did insert the foreign
nullifier).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 7c3563cd5a wallet/rpc: order history RPC txs by pointer, not by deep copy 
getalldata, z_listreceivedaddress and z_listsentbyaddress build a
map<int64_t, CWalletTx> of the whole wallet, keyed by nOrderPos, before
any filter runs. Every insert deep-copies the CWalletTx, including each
note's witness list (~100KB per note on a mainnet wallet). That is
gigabytes of allocation per call on a large wallet, under
LOCK2(cs_main, cs_wallet). The GUI's history scan runs
z_listsentbyaddress on every launch and about once per new tip.

The map now holds const CWalletTx* into mapWallet. It is still a
std::map, so the first tx inserted for an nOrderPos wins, exactly as
before. mapWallet can't change while the RPC holds cs_wallet, so the
pointers stay valid. Output is unchanged for a consistent wallet. One
corner case differs: with a nullifier map entry pointing at a note
its tx lacks, the spends helper's operator[] inserts an empty note into
that tx, and a later iteration now sees it where the old copies didn't.
The next commit removes that insertion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 190023f85f wallet: note-cache review follow-ups 
- UpdatedNoteData now keeps the incoming plaintext when it keeps the
  wallet's own entries. When it returns "unchanged" (equal note data,
  or SaplingNoteDataIdentical), any entry lacking a plaintext takes the
  incoming one, for the same outpoint and ivk only. The startup rescan
  replays loaded txs with FindMySaplingNotes' freshly decrypted data;
  that plaintext was thrown away, so the warm-up decrypted those notes a
  second time. The cache is outside every comparison, so the function
  still returns false and nothing is rewritten.
- -checknotecache is read once at init into fCheckNoteCache, instead of
  a mapArgs lookup on every GetFilteredNotes call. Its help text and
  comment now say what it checks: every note the walk considers, before
  the address filter, so a corrupt cached address can't drop a note
  unchecked.
- GetFilteredNotes pre-sizes its result only when there's no address
  filter, so a one-address call no longer reserves room for the whole
  wallet.
- WarmSaplingNoteCache returns its counts (notes, filled, no such
  output, undecryptable), and FillSaplingNoteCache records the threads
  it used, so tests can tell the cases apart.
- WarmSaplingNoteCache's comment no longer claims nothing can wait on
  cs_wallet while it runs: the RPCs allowed during warmup can.

Tests:
- ReplayWithPlaintextIsNoUpdate pins the rule that the plaintext cache
  stays out of the "did this tx change?" decision. A rescan replays a
  witnessed note loaded without plaintext, using FindMySaplingNotes'
  data that carries one. That must be no update: no rewrite, and the
  witness and nullifier are kept. The stored entry then gains the
  plaintext. Checked by mutation: with cache presence added to
  SaplingNoteDataIdentical, this test fails and every other test still
  passes.
- ParallelFillMatchesSerial uses 460 notes and asserts the thread count.
  So the 7-thread case really runs 7 threads, with uneven shards
  (460 % 7 != 0). Before, it was capped at 4 and repeated the 4-thread
  case.
- ColdParallelFillMatchesLegacy: a cold GetFilteredNotes over ~270
  candidates fills on 4 threads and matches the 69bb035e7 loop entry
  for entry.
- WarmUpSkipsBadNotes asserts the warm-up's separate counts (1 no such
  output, 2 undecryptable), so removing the out-of-range guard is now
  caught.
- CachedPlaintextMatchesFreshDecrypt checks exact counts.
  CheckNoteCacheRepairsADifference also tampers with a memo, and looks
  entries up by outpoint. The meaningless GetHash check is dropped.
- A ScopedArg / ScopedCheckNoteCache RAII helper restores -args and
  fCheckNoteCache even when an assertion returns early.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 23877e1038 wallet: -checknotecache, a diagnostic for the note plaintext cache 
With -checknotecache=1 (debug help), GetFilteredNotes decrypts every
note it returns again. If the cached plaintext differs from the fresh
decryption, it logs the note and replaces the cache entry with the
fresh value. It costs the full decryption the cache exists to avoid, so
it is off by default; it is there for validating the cache on a real
wallet, or for diagnosing a suspected cache problem in the field.

CheckNoteCacheRepairsADifference: by default a tampered cache entry is
returned as is (the cache is trusted); with the option it is repaired
and the fresh value is returned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 43635b7fd7 wallet: decrypt big note-cache batches on worker threads 
The startup warm-up, or a lazy fill after -notecachewarmup=0, decrypts
the whole wallet: ~0.5ms per note, ~6s serial for 12k notes. Each
decryption is independent: it reads its own output and writes only its
own note's cachedPlaintext. So FillSaplingNoteCache now splits batches
of 128 or more notes round-robin across -witnessbuildthreads (default:
hardware_concurrency) workers, at least 64 notes per thread. It follows
BuildWitnessCache's Phase 2 pattern:

- the caller keeps cs_wallet held throughout;
- workers take no lock and assert none; under DEBUG_LOCKORDER a fresh
  thread has no lock stack;
- per-shard fill counters are summed after join;
- nothing escapes a worker: a throwing decryption leaves that note
  unfilled, like one that doesn't decrypt;
- if the OS refuses a thread, the remaining shards run inline.

librustzcash is already called concurrently by BuildWitnessCache's
workers, and libsodium is thread-safe after sodium_init.

ParallelFillMatchesSerial: 300 notes across two keys and diversified
addresses decrypt to exactly the same cache with 1, 4 and 7 threads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 83a14a7bfb wallet/init: decrypt every wallet note into the plaintext cache at startup 
The plaintext cache is in memory only, so a restart empties it. Without
a warm-up, the first balance call after a restart decrypts the whole
wallet under LOCK2(cs_main, cs_wallet): ~6s for a 12k-note wallet. That
is the observed failure: restart, the GUI connects, and the node looks
stuck.

CWallet::WarmSaplingNoteCache decrypts every note whose plaintext isn't
cached, spent ones included, because z_listreceivedbyaddress
(ignoreSpent=false) is called once per address by the GUI's history
scan.

init runs it at the end of the wallet block, just before
SetBroadcastTransactions. At that point RPC is still in warmup (it
shows "Decrypting wallet notes...") and the wallet notification
thread, whose startup witness rebuild can hold cs_wallet for a long
time, has not started yet. It takes cs_wallet only. -notecachewarmup=0
(debug help) skips it, which leaves the lazy fill in GetFilteredNotes
to do the work.

Unlike GetFilteredNotes, the warm-up never asserts. A note that doesn't
decrypt under its ivk, or names an output its tx lacks, is counted in
the summary line and listed under -debug=zrpcunsafe. A corrupt note
therefore can't stop the node from starting, or from serving the
key-export RPCs allowed during warmup; GetFilteredNotes still asserts
on it, as before.

Tests:
- WarmUpFillsEveryNote: spent, locked, unconfirmed and stale-block
  notes included; walks afterwards decrypt nothing.
- WarmUpSkipsBadNotes: undecryptable and out-of-range notes are skipped
  without aborting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 b1e3c5230a wallet: cache each Sapling note's decrypted plaintext in memory 
GetFilteredNotes, the walk behind every balance and note-listing RPC,
sweep, consolidation, z_sendmany and z_mergetoaddress, decrypted every
note it looked at on every call: ~0.5ms of Jubjub arithmetic per note,
under LOCK2(cs_main, cs_wallet). A note's plaintext is fixed by its
output (the txid commits to the ciphertext) and the ivk that decrypts
it, so it never needs to be decrypted twice.

SaplingNoteData gains an in-memory-only
boost::optional<SaplingNoteCache> {note, memo}; the address is the
note's (d, pk_d). It follows the witnessRootValidated precedent:
- never serialized, and left out of operator== and
  SaplingNoteDataIdentical, so filling it never makes a tx look changed
  and get rewritten (the replay rewrite storm 69bb035e7 removed);
- FindMySaplingNotes fills it from the decryption that found the note
  (both the spending-key and the viewing-key-only loop);
- UpdatedNoteData's replace branch keeps a plaintext the wallet already
  had, for the same outpoint and ivk only;
- GetFilteredNotes now runs in two passes: pass 1 applies the tx,
  spent and locked filters and collects the notes that pass; whatever
  lacks a plaintext (everything after a restart, until the startup
  warm-up in a later commit) is decrypted once via the new
  FillSaplingNoteCache; pass 2 applies the address and spending-key
  filters and emits entries in the same order as before.
Spentness, spendability, lock state and depth are still decided on every
call; only the plaintext is cached.

The A0 equivalence test passes unchanged (warm). New tests:
- ColdCacheMatchesLegacy: the same grid with the cache dropped before
  every call;
- CachedPlaintextMatchesFreshDecrypt: eager and lazy fills;
- WarmPassDoesNotDecrypt: a fill counter shows a warm walk decrypts
  nothing, and spent or locked notes only when the caller asks for
  them;
- CacheNeverSerialized: byte-identical SER_DISK/NETWORK/GETHASH output,
  empty after a load;
- UpdatedNoteDataAndTheCache: no rewrite on a cache-only difference,
  carried on replace, never across another ivk;
- SpendabilityNotCached;
- FindMySaplingNotesFillsCache.

Memory: ~616 bytes per note (the memo is 512), ~8MB for a 13k-note
wallet, next to ~100KB per note of cached witnesses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 7ddd03d86d zcash/wallet: decrypt a Sapling note's plaintext and note in one pass 
SaplingNotePlaintext::decrypt(ciphertext, ivk, epk, cmu) derives
pk_d = [ivk]g_d to check the note commitment, then throws it away. The
wallet then calls ivk.address(d), which derives pk_d again, and
plaintext.note(ivk), which derives it a third time. Each derivation is a
Jubjub scalar multiplication, ~115us, measured. Including the key
agreement and the commitment check, the whole sequence is ~775us per
note.

decrypt_with_note() runs the same decryption and checks and returns the
note built from that pk_d. ivk_to_pkd succeeding already implies a
valid diversifier, which is all ivk.address() checks on top. It gets its
own name rather than a decrypt() overload: an overload taking a fifth
argument by non-const reference would capture calls meant for the
(epk, esk, pk_d, cmu) variant. The cmu check stays.

GetFilteredNotes and FindMySaplingNotes' spending-key loop use it. That
saves two of the three derivations per note in every balance walk, and
one per found note during scans and rescans (trial decryption of
outputs that aren't ours still fails at the AEAD, as before).
FindMySaplingNotes also no longer copies each OutputDescription.

New tests: MatchesDecryptThenNote checks decrypt_with_note field for
field against decrypt() + note(ivk), over default and diversified
addresses, a wrong ivk and a wrong cmu. FindMySaplingNotes pins what
discovery returns: notes per key, diversified addresses to register,
foreign outputs ignored. Both pass before and after the switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 bfec5c74ae gtest: notecache review fixes 
- Take LOCK2(cs_main, cs_wallet) before adding keys in SetUp: both
  AddSaplingZKey and AddSaplingIncomingViewingKey assert cs_wallet is
  held, which aborts a DEBUG_LOCKORDER build.
- TIP_HEIGHT becomes a namespace-scope constexpr. As a static class
  member it was ODR-used by EXPECT_EQ without a definition, which fails
  to link as C++11 at -O0.
- TearDown restores the previous chainActive tip and erases only the
  mapBlockIndex entries the fixture inserted.
- Add a tx mined only in a stale side block. The legacy tx_height model
  returns that block's height, as a txindex can, and the tx must still
  be excluded at every minDepth.
- The encrypted variant asserts HaveSaplingSpendingKey directly, and a
  new test unlocks the keystore and checks GetSaplingSpendingKey (the
  other loop the crypter change touched).
- The legacy-loop comment now says its two LogPrintf lines are dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 21586b353a wallet/rpc: monotonic timers and a quieter default slow-walk line 
Review follow-ups to the GetFilteredNotes / RPC timing logs:

- Time with std::chrono::steady_clock instead of GetTimeMicros(), which
  is the system clock. A wall-clock step otherwise writes false
  ">= 500ms" lines unconditionally; on a WSL2 host the clock was seen
  stepping +3.6s every ~32s.
- A long-poll getblocktemplate deliberately blocks (with cs_main
  released) until a new block arrives, so it is logged only under
  -debug=bench.
- Log the method as "RPC method=<name> took <n>ms". A bare " rescan "
  token followed by "<n>ms" matches ObsidianDragon's rescan-finished
  log heuristic.
- The always-on slow-walk line keeps just the filters and the walk
  time; the wallet-size counts (txs, notes, spent, entries) are logged
  only with -debug=bench, so a shared debug.log doesn't carry them.
- The -txindex=0 startup line names the version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 34b94e761a wallet: don't copy every encrypted Sapling key on each lookup 
CCryptoKeyStore::HaveSaplingSpendingKey and GetSaplingSpendingKey scan
mapCryptedSaplingSpendingKeys with `for (auto entry : ...)`. That copies
each entry (an extended full viewing key plus its encrypted-secret
vector) for every key visited, on every call. GetFilteredNotes'
requireSpendingKey check makes one such call per note, and the history
RPCs make one per output per address: ~22ns per key per call, measured
(11us per note on a 1024-key wallet). Iterate by const reference.

The equivalence test gains a locked, encrypted-keystore variant,
LockedWalletMatchesLegacy, so spendability coming from the encrypted
map is covered too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 94967efa7b wallet: take note heights from the wallet's depth, not tx_height() 
For minDepth > 1, GetFilteredNotes located each wallet tx's block with
tx_height(), which calls GetTransaction and re-reads the tx from disk
through the txindex. It did this once per wallet tx, transparent ones
included, while holding LOCK2. z_listunspent and z_listreceivedbyaddress
did it once more per returned note. Without -txindex, GetTransaction can
only find a tx that still has unspent transparent outputs. tx_height()
then returned 0 for every fully shielded tx, and the tx was skipped. So
on a -txindex=0 node, sweep and consolidation (minDepth 11) and any
minconf >= 2 call never saw those notes.

The wallet already has the depth, and for an active-chain tx
GetDepthInMainChain is chainActive.Height() - height + 1. The new
HeightFromDepth() inverts that (0 when unconfirmed or off the active
chain) and replaces tx_height() at all three sites. The height only
feeds the == 0 skip and hush_dpowconfs. Both sides hold cs_main for the
whole call, so the tip can't move between the walk and the RPC tail.

On a -txindex node the results are identical: the txindex block and
wtx.hashBlock are the same block. The extended equivalence test now
also covers minDepth 2/11/12/22. On -txindex=0 nodes, fully shielded
notes now show up for minDepth > 1. With -zsweep or -consolidation
enabled, those notes will now be swept and consolidated, so init logs
that once. MinDepthAboveOneUsesWalletHeight pins the new behaviour and
shows that the old path returned nothing without a txindex.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 bfc1334565 wallet: walk GetFilteredNotes by reference, cheap filters first 
GetFilteredNotes copied every CWalletTx by value before any filter ran,
then copied each SaplingNoteData by value again. Both copies deep-copy
the note's witness list (up to WITNESS_CACHE_SIZE witnesses, ~100KB per
note on a mainnet wallet): ~40us per note per pass, measured. Spent and
locked notes were also fully decrypted (~0.6ms of Jubjub arithmetic per
note) before the spent and locked checks threw them away. -deletetx is
off by default, so spent notes accumulate.

- iterate mapWallet and mapSaplingNoteData by const reference;
- skip txs with no notes of ours before CheckFinalTx;
- read GetDepthInMainChain once per tx instead of 2+N times (cs_main is
  held, so it can't change during the walk);
- run the spent and locked checks before the decryption;
- z_listreceivedbyaddress no longer copies the whole CWalletTx per
  entry to read its time, and z_sendmany's sort comparator takes its
  entries by reference.

The filters have no side effects, so the result set and its order are
unchanged; the A0 equivalence test passes unchanged. One deliberate
difference: a spent or locked note whose ciphertext no longer decrypts
under its ivk is now skipped instead of tripping the decrypt assert. The
new SpentAndLockedNotesAreNotDecrypted test pins this (it aborts on the
old ordering).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 b9cdce8e96 wallet/rpc: time GetFilteredNotes walks and slow RPCs 
GetFilteredNotes logged "GetFilteredNotes ignoreLocked=N" on every call.
The balance RPCs run it on every GUI poll, so that line was log spam on
a fast wallet, and on a slow one the gaps between its timestamps were
the only timing there was. Those gaps also count lock waits and the
previous RPC's own post-processing.

It now logs one summary per walk: the filters, txs that passed the
tx-level checks out of the wallet total, notes examined, notes skipped
as spent, entries returned, and the walk time. The summary is always
written when the walk takes >= 500ms (that long under cs_main stalls
block connection) and otherwise only with -debug=bench. The line has no
lock-wait figure: every caller already holds LOCK2(cs_main, cs_wallet)
and CCriticalSection is recursive, so the inner LOCK2 never waits.

Lock waits are measured where they happen instead. CRPCTable::execute
logs "RPC <method> took <n>ms" under the same >= 500ms / -debug=bench
rule. The per-locked-note line moves to -debug=zrpcunsafe, since it
logs txid prefixes.

No behaviour change otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 dc8ca4996a gtest: pin GetFilteredNotes behaviour before optimizing it 
Adds test_wallet_notecache.cpp: a wallet with Sapling notes at several
depths (spent by a mined tx, spent by an unmined tx, locked, viewing-key
only, never mined, plus txs carrying no wallet notes), mined into a fake
active chain. The fixture undoes everything it puts into the process-wide
chain state (chainActive, mapBlockIndex, pcoinsTip) in TearDown, since all
gtest suites share one process.

LegacyFilteredNotes is the 69bb035e7 loop verbatim, with tx_height()
passed in so the minDepth > 1 branch can be modelled on a -txindex node.
FilteredNotesMatchLegacy compares the real walk against it entry by entry
and in order (z_sendmany sorts the result with an unstable sort) over
minDepth {0,1} x maxDepth x ignoreSpent x requireSpendingKey x
ignoreLocked x address filters. FixtureExercisesEveryFilter checks that
each filter really removes something, so the comparison can't pass
vacuously.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 b911189ac6 wallet: don't witness viewing-key-only notes (fixes an abort on the next block) 
A note found through an incoming viewing key only (z_importviewingkey)
has no full viewing key behind it. UpdateSaplingNullifierNoteMapWithTx
skips such notes, so they never get a nullifier. VerifyAndSetInitialWitness
treated "no nullifier" as "witness needs reseeding" on every call: it
cleared the note's witness, re-read the block from disk, reseeded it,
and then dereferenced the nullifier that never came. dragonxd aborted
in the txnotify thread (boost::optional "is_initialized" assert) on the
first block confirming a note to such an address, and again after every
restart, before the block index was flushed.

Such a note can't be spent, so it needs no witness. Skip it there; if
its spending key is imported later, it has a full viewing key and the
existing no-nullifier rule seeds it then. SaplingWitnessMinimumHeight
now takes the optional nullifier. A note without one is never extended
by BuildWitnessCache (its pending-note gate requires one), so it must
not lower the rebuild's start height. Every call site that worked before
passes a set nullifier and behaves as before.

The deref came in with the 2020 witness rework (6c48397cf). The new
gtest ViewingKeyOnlyNoteIsNotWitnessed fails on the old code (the note
reaches the reseed path).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-17 08:16:09 -05:00
DanSandClaude Opus 5.5 7131b3673f qa: add fd-guard, the close() guard and regtest harness that found the double close 
shim/ is an LD_PRELOAD guard around close/fclose/dup2/dup3:
- before each close it resolves /proc/self/fd/N;
- it reports, with a backtrace, a non-owner close of wallet.dat,
  database/log.* or blocks/*.dat, and every close that fails with EBADF
  (the second half of a double close);
- FDGUARD_ABORT=1 aborts on the first hit;
- run_tests.sh self-tests it (51 checks).

repro/ runs a shimmed dragonxd on isolated regtest. Each run gets its
own unprivileged network namespace (loopback only). The loads are:
- rpc: a hammer with impatient/RST/half-open clients, work-queue
  bursts and long-polls;
- p2p: TLS peers churned by disconnectnode, ban and kill, plus a raw
  TCP/TLS misbehaver;
- tor: a fake control port;
- mine: mining with blocknotify and autoshield.
tools/tally.py groups the reports by closer. tools/scan_bdb.py finds BDB
pages written into block files.

With the wolfSSL CloseSocket macro in effect:
- p2pmin (300 s) gives one EBADF double close per disconnect;
- the full load (1200 s) gives 930 EBADF closes, 186 non-owner closes
  (37 of wallet/log/blk files) and a 632 KB db.log with BDB0060 PANIC;
- RPC-only and Tor-only loads give none.
After the fix, every count is 0 and db.log stays empty. The README has
the commands and the numbers.

Paths come from the environment (GOLDEN, BIN, RUNS, SHIM, ASMAP and the
Sapling params). Run output goes to $RUNS, outside the tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 21:23:01 -05:00
DanSandClaude Opus 5.5 70b35ac11a wallet/db: keep the BDB logs when the shutdown flush fails 
CDBEnv::Flush(true) removed database/ (log_archive DB_ARCH_REMOVE, then
remove_all) once mapFileUseCount was empty. It did that even when the
checkpoint or lsn_reset before it had failed, or the environment had
panicked (BDB0060). In those states wallet.dat can be missing committed
changes that only the logs hold:
- a page write failed (EBADF/ESPIPE through an fd closed behind BDB's
  back, the wolfSSL CloseSocket bug), so the page stayed dirty in the
  mpool cache and the checkpoint never completed;
- pages written after the last lsn_reset carry real LSNs that point
  into those logs.
Deleting the logs turned a wallet that the next start's DB_RECOVER would
repair into one that silently lost those changes. A std::terminate kept
the logs; the clean shutdown that ThreadNotifyWallets now does on a
wallet DB error deleted them.

Remember a failed txn_checkpoint or lsn_reset (in Flush and in
CheckpointLSN, which the periodic flush and backupwallet use). Before
removing anything, also check the environment for a panic. get_flags
reports it, and itself fails with DB_RUNRECOVERY once the environment
has panicked. In any of those cases keep database/ and log why. If the
kept logs are unusable, the next start already moves database/ aside
and retries (CWallet::Verify).

WalletDbEnv tests:
- a clean shutdown still removes the logs;
- a panicked environment (DB_PANIC_ENVIRONMENT) keeps them;
- a failed lsn_reset keeps them.
With the old db.cpp only the first passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 21:23:01 -05:00
DanSandClaude Opus 5.5 5b36d7ad24 validationinterface: shut down cleanly when a wallet block notification throws 
ThreadNotifyWallets' block disconnect and connect loops had no
try/catch (only its mempool loop does). A wallet database failure
inside SyncWithWallets or ChainTip escaped the thread, and TraceThread
logs and rethrows, so the node died in std::terminate with nothing
flushed. One example: CWalletDB::ListAccountCreditDebit's "error
scanning DB" after BDB lost its file to the wolfSSL CloseSocket bug.
That is how the pool's node died on 2026-10-12 and 2026-10-16.

Run each block's wallet notifications through NotifyWalletsOrShutdown.
On an exception it logs the error, requests a shutdown, shows the usual
fatal-error message and stops the notifier, leaving pindexLastTip at
the last block the wallet fully processed. Carrying on instead would
leave the wallet behind the chain, missing the block it failed on.
Thread interruption still propagates. The shutdown is requested before
the fatal-error message is shown.

FdDefense.WalletNotifyFailureShutsDownCleanly:
- a throwing notification requests a shutdown, a clean one doesn't;
- a non-std exception is handled the same way;
- boost::thread_interrupted still propagates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 21:23:01 -05:00
DanSandClaude Opus 5.5 9cb31326e3 main: open block and undo files read-only when they're opened for reading 
OpenDiskFile opened every block or undo file "rb+", even for its
fReadOnly callers, so every reader held a writable fd. That's what let
the wolfSSL CloseSocket bug corrupt blocks/*.dat. A P2P double close
took the wallet's BDB fd away; the number was then handed to a block
file opened for a getblock/getdata read; BDB's next wallet page write
landed in it, at offset pgno*4096. On production that destroyed blocks
2,728,132-2,728,161 in blk00011.dat and block 3,356,880 in blk00020.dat.

Every fReadOnly caller only reads through the handle:
- ReadBlockFromDisk, UndoReadFromDisk, both GetTransaction paths and
  hush_blockload read via CAutoFile >>;
- the reindex loop hands it to LoadExternalBlockFile, which reads via
  CBufferedFile;
- the existence check at startup only opens it;
- hush_prefetch only freads.
Open those "rb". A stray write through a reused fd number then fails
(EBADF) instead of silently corrupting a block file. Missing files
still fail to open as before; writers keep "rb+"/"wb+".

FdDefense.ReadOnlyBlockFileIsNotWritable: pwrite through a read-only
block file's fd fails with EBADF, while the writable open still writes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 21:23:01 -05:00
DanSandClaude Opus 5.5 d91e069da3 net: stop wolfSSL's CloseSocket macro from turning P2P closes into double closes 
wolfssl/wolfio.h (included by <wolfssl/ssl.h> in net.h) does
`#ifndef CloseSocket / #define CloseSocket(s) close(s)`. Every TU that
includes net.h after that point compiles CloseSocket(x) to a bare
close(x). That includes all 15 calls in net.cpp, among them the ones
in CloseSocketDisconnect and ~CNode. util/sock.cpp's
CloseSocket(SOCKET&), the only thing that sets the handle to
INVALID_SOCKET, was never called from net.cpp. This has been the case
since OpenSSL was replaced by wolfSSL (Hush 9ff756590, 2020), so every
DragonX release is affected.

So CNode::hSocket kept its number after the socket was closed. Every
P2P disconnect closed that same number again, 2 to 4 times:
- CloseSocketDisconnect (net.cpp:572) runs once per disconnect trigger
  (ThreadSocketHandler, SocketSendData, the TLS socket handler,
  ThreadMessageHandler), and each run still saw a valid-looking hSocket;
- ~CNode (net.cpp:2819) then closed it once more.
Meanwhile the socket handler kept FD_SETting, send()ing and recv()ing on
the stale number. In between, other threads get that number from the
kernel's lowest-free-fd rule:
- BDB reopening wallet.dat or database/log.* (ThreadFlushWalletDB, ~11/s);
- fopen("rb+") of a block file for getdata/getblock;
- an HTTP RPC accept.
The stray close then closes it. BDB keeps doing I/O on the dead number:
'wallet.dat: write failed for page N' / 'Illegal seek' / 'BDB0060
PANIC', or silent wallet-page writes into a blocks/blk*.dat that got
the number next. Its failing reads later threw out of ThreadNotifyWallets
(std::terminate). A closed HTTP connection fd makes libevent re-arm
and later close whatever owns that number next ('Epoll ADD ... failed').

Reproduced on isolated regtest with an LD_PRELOAD close() guard:
- one disconnectnode gives exactly one EBADF double close (49
  disconnects, 50 EBADF; with strace and no shim, 29 disconnects gave
  30 'close(N) = -1 EBADF');
- P2P churn alone gives non-owner closes of block files and wallet.dat;
- RPC-only and Tor-only loads give none.
Disassembly of CloseSocketDisconnect and ~CNode shows 'call close@plt'
before this change and 'call CloseSocket(unsigned int&)' after.

Undefine the macro right after the wolfSSL include in net.h; wolfio.h
is include-guarded, so it can't come back. net.cpp gets an #error
tripwire in case it ever does. NetCloseSocket.ClosedNodeNeverClosesAReusedFdNumber
fails on the old code (a closed CNode closes the fd number's new owner)
and passes now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-16 21:23:00 -05:00
DanSandClaude Opus 5.5 69bb035e7e wallet/main: break the replay -> witness-rebuild -> kill loop on large wallets 
A v1.3.0 node with a ~13.6k-note wallet (-dbcache=4084) looked hung for hours:
every start replayed ~62k blocks from a chainstate 26 days stale, then ran a
47-60 min all-core witness rebuild holding LOCK2(cs_main, cs_wallet) (RPC answered
HTTP 500 "Work queue depth exceeded", peers timed out), the GUI killed it before
anything was flushed, and the next start did it all again. Even after escaping,
each new block cost ~30s under LOCK2, so the node fell behind a 36s chain.

D1  BuildWitnessCache: the post-rebuild latch computed witnesses.front().root()
    (~2.4ms) for every pending note on every block, only to feed a log line that
    is printed for bulk rebuilds alone. The anchor check now runs on the Phase-2
    workers and only for a bulk rebuild; the per-block path just sets the latch.
    Phase 2 also runs in parallel whenever the job is big (notes x commitments
    >= 2000), not only for a bulk range, so busy blocks no longer extend every
    note serially. If the OS refuses a worker thread, the remaining shards run
    inline instead of unwinding past joinable threads (std::terminate).

D2  FlushStateToDisk: the coins cache was only flushed on shutdown, gettxoutsetinfo
    or after 24h; on this chain it stays a few MiB, so the size triggers never
    fire and a node that never reached 24h uptime or a clean stop kept its
    chainstate (and block index) weeks behind. Flush at least hourly and every
    DATABASE_FLUSH_BLOCKS (5000) connected blocks -- the latter also inside one
    long ActivateBestChain -- and right after the startup ActivateBestChain when
    it moved the tip.

D3  UpdatedNoteData: a replayed or rescanned tx arrives with FindMySaplingNotes
    data (no nullifier). Witnesses and witnessHeight were carried over but the
    nullifier was dropped, so VerifyAndSetInitialWitness cleared and reseeded the
    witnesses on the next block -- the replay above rewrote 3,750 txs and forced
    the full rebuild. Carry the nullifier with the witnesses, unless the tx was
    re-mined in a different block (its notes moved; reseed as before). When the
    merged data is identical, don't rewrite the tx; just clear the in-memory
    witnessRootValidated so the next block re-checks the kept roots once (a
    rescan still finds and reseeds a witness that went bad, without rebuilding
    the healthy ones). The carry loop no longer deep-copies every note.

    SetBestChain: with hourly coin flushes the wallet locator must not run ahead
    of the async notifier, or a crash permanently skips the blocks it had not yet
    delivered. Stamp the block the wallet's state reflects (pindexWalletTip,
    seeded in init before ThreadNotifyWallets starts -- at the startup rescan's
    checkpoint if that rescan was interrupted -- and advanced contiguously by
    ChainTip), like zcashd's ChainTipAdded. ThreadNotifyWallets no longer hands
    the wallet an empty block after a fatal block-read failure.

Tests: new gtest test_wallet_notedata.cpp (5 tests); 23/23 gtests pass, and the
two core D3 tests fail with D3 reverted. Regtest A/B against master with a
1,008-note wallet: notify-thread CPU per block 1,780ms -> 3ms; worst LOCK2 RPC
stall 1.6-2.9s -> 10-15ms on ordinary blocks and 4.6-7.1s -> ~1.6s on a block
carrying a 40-output self-send; a z_importkey rescan went from 35 reseeds plus a
bulk rebuild (10.6s) to 0 reseeds, same balance; after kill -9 at 150 blocks a
build with a 100-block flush threshold restarted at 100, master at 0. Two
adversarial review rounds; every confirmed finding is addressed here.

Not covered: the hourly SetBestChain/flush paths were reviewed, not exercised
(GetTimeMicros ignores mocktime); no run yet against a copy of a real large wallet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-09-30 19:05:45 -05:00
DanSandClaude Opus 5 da191e447d Merge branch 'dev' for the v1.3.0 release 
72 commits since 1c3523aac (2026-07-23). Highlights:

  consensus  BLOCK_VALID_CONTEXT no longer overwrites the block validity level
             (6d282db21); dead CBOPRET coinbase price check removed (267e6f7ad);
             PROTOCOL_VERSION 2000000 -> 2000001, MIN_PEER_PROTO_VERSION held.
  sync       DRAGONX checkpoints extended to 3,226,000 -- past the RandomX
             activation at 2,838,976, so the skip-below-checkpoint path finally
             fires (7dc904c96).
  wallet     interrupted rescans no longer hide funds and no longer latch
             fAbortRescan (a6b6f80db); rescan progress is checkpointed when a scan
             is interrupted (c1040028e); salvaged wallets open in a degraded mode
             with hdchain-mismatch detection (5634aed75).
  stratum    stratummine gated behind an explicit flag and hardened (db42091ce);
             low-difficulty shares rejected before spending a RandomX hash
             (fa16e740b).
  release    version bumped to v1.3.0 (af7d9e230) and doc/release-process.md
             corrected -- it previously omitted signing entirely, which is why
             v1.1.0 and v1.2.0 were tagged but never became installable releases.

Soak before merge: ~86 node-hours on c1040028e across all seven seeds, plus the
primary -- the pool's block-template source -- running it in production with the
pool accepting shares. Fleet consensus verified at a common height throughout.

Not included, deliberately: branch audit-fixes-20260901 (two commits, both marked
UNTESTED) stays parked. The ported rpc-tests have never been run green and are not
a gate for this release.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
 2026-09-01 19:09:04 -05:00
DanSandClaude Opus 5 fc26e2e0a0 doc: stamp the man pages with the release version, not a build sha 
The man pages were regenerated at e2e10f6ef from a binary built at af7d9e230 and
carry "v1.3.0-af7d9e230". The release will be tagged v1.3.0, so the shipped man
pages would contradict the tag they ship under -- the same provenance gap af7d9e230
itself was written to close.

A faithful regeneration cannot fix this before the tag exists: gen-manpages.sh reads
--version from the binary, genbuild.sh derives that from `git describe`, and that
only prints exactly "v1.3.0" once the annotated tag is in place. Regenerating after
tagging would mean committing on top of the tag and moving it. doc/release-process.md
anticipates this and documents hardcoding the version instead.

Content is untouched and needs no regeneration: all 150 daemon options in the man
page match `dragonxd --help` exactly, including the five -autoshield* flags.

Not changed: dragonx-tx.1 still describes itself as "hush-tx utility". That string
comes from the binary's own --help, so editing the man page would make it diverge
from what the tool actually prints. It is a binary-level branding fix, not a doc one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
 2026-09-01 19:06:54 -05:00
DanSandClaude Opus 5 1d2d5f32d0 version: bump PROTOCOL_VERSION for the v1.3.0 consensus changes 
doc/release-process.md requires PROTOCOL_VERSION to increase by 1 for any release
carrying a consensus change. This one does: 6d282db21 stops BLOCK_VALID_CONTEXT
overwriting the block validity level, 267e6f7ad drops the dead CBOPRET price
validation from the coinbase check, and pow.cpp is 157 lines lighter. The value
has not moved since 85c8d7f7d in March.

MIN_PEER_PROTO_VERSION stays at 2000000. It currently equals PROTOCOL_VERSION, so
raising it in step would disconnect every peer still announcing 2000000 -- which is
every user, because no DragonX release has ever been installable: the wallet's
updater pins an ed25519 key and sets kDaemonRequireSignature = true, and no release
has ever shipped a .sig. Bumping only the advertised version lets 1.3.0 nodes be
identified on the wire without partitioning the network on release day.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
 2026-09-01 19:06:54 -05:00
46 changed files with 6848 additions and 219 deletions
Expand all files Collapse all files
doc/man/dragonx-cli.1
+1 -1
View File
@@ -3,7 +3,7 @@
.SH NAME
dragonx-cli \- manual page for dragonx-cli v1.3.0
.SH DESCRIPTION
DragonX RPC client version v1.3.0\-af7d9e230
DragonX RPC client version v1.3.0
.PP
In order to ensure you are adequately protecting your privacy when using
DragonX, please see <https://dragonx.is/security/>.
doc/man/dragonx-tx.1
+1 -1
View File
@@ -3,7 +3,7 @@
.SH NAME
dragonx-tx \- manual page for dragonx-tx v1.3.0
.SH DESCRIPTION
hush\-tx utility version v1.3.0\-af7d9e230
hush\-tx utility version v1.3.0
.SS "Usage:"
.TP
hush\-tx [options] <hex\-tx> [commands]
doc/man/dragonxd.1
+1 -1
View File
@@ -3,7 +3,7 @@
.SH NAME
dragonxd \- manual page for dragonxd v1.3.0
.SH DESCRIPTION
DragonX Daemon version v1.3.0\-af7d9e230
DragonX Daemon version v1.3.0
.PP
In order to ensure you are adequately protecting your privacy when using
DragonX, please see <https://dragonx.is/security/>.
qa/fd-guard/.gitignore
+4
View File
@@ -0,0 +1,4 @@
shim/libfdguard.so
shim/fdguard_test
*.o
shim/t/
qa/fd-guard/README.md
+55
View File
@@ -0,0 +1,55 @@
# fd-guard: catch stray close() of files a thread doesn't own
This is the harness that found the wolfSSL `CloseSocket` macro bug. With that bug, P2P disconnects closed their fd number a second time, after another thread had reused it, and BDB then wrote wallet pages into `blocks/*.dat`.
It has two parts:
- `shim/`: an `LD_PRELOAD` guard around `close`, `fclose`, `dup2` and `dup3`.
- Before each close it resolves `/proc/self/fd/N`.
- It logs a backtrace when a non-owner closes `wallet.dat`, `database/log.*` or `blocks/*.dat`, and for every close that fails with `EBADF` (the second half of a double close).
- `FDGUARD_ABORT=1` makes the first hit abort.
- `repro/`: isolated regtest runs that put a shimmed `dragonxd` under load:
- **rpc:** impatient and RST clients, half-open connections, work-queue bursts and long-polls;
- **p2p:** TLS peers churned by `disconnectnode`/ban/kill, plus misbehaving raw TCP/TLS clients;
- **tor:** a fake control port;
- **mining:** with `blocknotify` and autoshield.
Each run executes in its own unprivileged network namespace: loopback only, no route.
## Requirements
- Linux.
- `unshare -rn` (unprivileged user namespaces), `nsenter`, `ip`, python3, gcc, and `addr2line` for symbolizing.
- A `dragonxd`/`dragonx-cli` built with `-g`. Adding `-rdynamic` makes backtraces readable without `addr2line`.
- A regtest datadir with a grown wallet to copy from (`GOLDEN`). It is never modified.
## Running
```sh
qa/fd-guard/shim/build.sh # builds libfdguard.so
qa/fd-guard/shim/run_tests.sh # self-test: double close, foreign close of a guarded file
cd qa/fd-guard/repro
export GOLDEN=/path/to/regtest/datadir BIN=/path/to/build/src # RUNS defaults to ~/fd-guard-runs
LOADS=p2pmin,mine DISC_INT=5 GUARD=all ./go.sh p2pmin 300 # 1 peer, disconnectnode every 5 s
LOADS=rpc,p2p,tor,mine WQ=64 RT=8 GUARD=all ./go.sh full 1200 # everything, production-like work queue
python3 tools/tally.py "$RUNS/<name>/fdguard.log" # who closed what
python3 ../shim/symbolize.py --summary "$RUNS/<name>/fdguard.log"
python3 tools/scan_bdb.py "$RUNS/<name>"/target/regtest/blocks/*.dat # BDB pages inside block files
```
Each run writes `run.txt` (full log plus summary), `fdguard.log` and the target's `debug.log` and `db.log` under `$RUNS/<name>/`.
## What to expect
Measured on a 2041-note regtest wallet, before and after the fix (`net: stop wolfSSL's CloseSocket macro ...`), under the same loads:
| Run | Binary | EBADF closes | Non-owner closes (wallet/log/blk) | libevent `Epoll` failures | `db.log` |
|---|---|---|---|---|---|
| `p2pmin`, 300 s | before | 51 (≈ 1 per disconnect) | 1 | 0 | empty |
| | after | 0 | 0 | 0 | empty |
| `full`, 1200 s, `WQ=64 RT=8` | before | 930 | 186 (37) | 111 | 632 KB: 7500 × `BDB0060 PANIC`, `write failed for page N` |
| | after | 0 | 0 | 0 | empty |
- RPC-only and Tor-only loads give 0 reports, even before the fix: P2P disconnects are the only origin.
- `scan_bdb.py` found no BDB pages in block files in any run. The landing of a page in a block file needs the reused number to be a writable block file at the moment BDB writes. Production did hit that (blk00011/blk00020), but it is rare.
- `SSL_write ... closing connection` lines are expected under `tcpchurn`. They are how the load triggers disconnects, not a symptom.
qa/fd-guard/repro/common.sh
+72
View File
@@ -0,0 +1,72 @@
# shellcheck shell=bash
# Shared config + helpers for the fd-double-close REPRODUCTION harness.
# Regtest only. Meant to run INSIDE an unprivileged network namespace (see ns.sh),
# so every port is private to the namespace and nothing can reach a real network.
set +e
REPRO=${REPRO:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)}
SRCROOT=$(cd "$REPRO/../../.." && pwd)
BIN=${BIN:-$SRCROOT/src} # dir with dragonxd + dragonx-cli (build with -g -rdynamic for backtraces)
DRAGONXD=$BIN/dragonxd
CLI=$BIN/dragonx-cli
RUNS=${RUNS:-$HOME/fd-guard-runs} # run output (block files, logs): keep it out of the repo
GOLDEN=${GOLDEN:?set GOLDEN to a regtest datadir with a grown wallet (it is only ever copied)}
ASMAP=${ASMAP:-$SRCROOT/asmap.dat}
SP=${SP:-$HOME/.zcash-params/sapling-spend.params}
OP=${OP:-$HOME/.zcash-params/sapling-output.params}
SHIM=${SHIM:-$REPRO/../shim/libfdguard.so}
RPCUSER=ab
RPCPASS=abpass
RPC_T=18901 ; P2P_T=18911
RPC_H1=18902; P2P_H1=18912
RPC_H2=18903; P2P_H2=18913
TORPORT=18950
clin() { local dd=$1 rp=$2; shift 2; "$CLI" -regtest -datadir="$dd" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$rp "$@"; }
write_conf() { printf 'rpcuser=%s\nrpcpassword=%s\nrpcport=%s\n' "$RPCUSER" "$RPCPASS" "$2" > "$1/DRAGONX.conf"; }
# prep_golden <dd> : copy of the golden wallet datadir (2041 notes, txindex=1, height 286)
prep_golden() {
local dd=$1
rm -rf "$dd"; cp -a "$GOLDEN" "$dd"
cp -f "$ASMAP" "$dd/asmap.dat"; cp -f "$SP" "$dd/sapling-spend.params"; cp -f "$OP" "$dd/sapling-output.params"
write_conf "$dd" "$RPC_T"
: > "$dd/regtest/debug.log"; rm -f "$dd/regtest/db.log"
}
# prep_fresh <dd> <rpcport> : empty regtest datadir (helpers)
prep_fresh() {
local dd=$1 rp=$2
rm -rf "$dd"; mkdir -p "$dd"
cp -f "$ASMAP" "$dd/asmap.dat"; cp -f "$SP" "$dd/sapling-spend.params"; cp -f "$OP" "$dd/sapling-output.params"
write_conf "$dd" "$rp"
}
start_helper() { # <dd> <p2p> <rpc> <logfile>
local dd=$1 p2p=$2 rp=$3 lg=$4
( cd "$dd" && exec nice -n 10 "$DRAGONXD" -regtest -datadir="$dd" -testnode=1 -checkpoints=0 \
-listen=1 -bind=127.0.0.1 -port=$p2p -connect=127.0.0.1:1 -addnode=127.0.0.1:$P2P_T \
-dnsseed=0 -dns=0 -asmap="$dd/asmap.dat" -autoshield=0 \
-rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$rp -listenonion=0 -maxinboundfromip=100 \
-debug=net -debug=tls -daemon ) >> "$lg" 2>&1
}
waitrpc() { # <dd> <rpc> [secs]
local dd=$1 rp=$2 n=${3:-300} i
for i in $(seq 1 "$n"); do clin "$dd" "$rp" getblockcount >/dev/null 2>&1 && return 0; sleep 1; done
return 1
}
stop_daemon() { # <dd> <rpc> (helpers, started with -daemon)
local dd=$1 rp=$2 pid i
pid=$(cat "$dd/regtest/hushd.pid" 2>/dev/null)
clin "$dd" "$rp" stop >/dev/null 2>&1
for i in $(seq 1 90); do { [ -z "$pid" ] || ! kill -0 "$pid" 2>/dev/null; } && return 0; sleep 1; done
[ -n "$pid" ] && kill -9 "$pid" 2>/dev/null
}
MARKERS=('libevent: Epoll' 'Socket close failed' 'Bad file descriptor' 'Operation not permitted' 'Illegal seek'
'SSL_write' 'Unable to seek' 'socket select error' 'write failed for page' 'PANIC' 'error scanning DB'
'terminate' 'Assertion' 'EXCEPTION')
qa/fd-guard/repro/go.sh
Executable
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
# go.sh <name> <seconds> -- create a private network namespace (loopback only, no route),
# run run.sh inside it, then tear the namespace down. Env knobs pass through to run.sh.
export REPRO=${REPRO:-$(cd "$(dirname "$0")" && pwd)}
export RUNS=${RUNS:-$HOME/fd-guard-runs}
OUT=$RUNS/$1; mkdir -p "$OUT"
setsid unshare -rn sh -c "ip link set lo up && echo \$\$ > $OUT/ns.pid && exec sleep infinity" </dev/null >/dev/null 2>&1 &
for _ in 1 2 3 4 5 6 7 8 9 10; do [ -s "$OUT/ns.pid" ] && break; sleep 0.3; done
NSP=$(cat "$OUT/ns.pid")
cp "$REPRO/run.sh" "$OUT/run.sh.snapshot" # run a frozen copy: editing run.sh cannot disturb a live run
nsenter -t "$NSP" -U -n --preserve-credentials bash "$OUT/run.sh.snapshot" "$@" > "$OUT/go.txt" 2>&1
rc=$?
kill "$NSP" 2>/dev/null
echo "go.sh: run.sh rc=$rc, namespace holder $NSP killed" >> "$OUT/go.txt"
qa/fd-guard/repro/run.sh
Executable
+156
View File
@@ -0,0 +1,156 @@
#!/usr/bin/env bash
# run.sh <name> <seconds> -- run INSIDE the network namespace (go.sh does that).
# Env knobs:
# LOADS=rpc,p2p,tor,mine which load generators run (default all four); p2pmin = 1 helper + disconnectnode every DISC_INT s
# WQ=4 RT=2 -rpcworkqueue / -rpcthreads of the target
# GUARD=all FDGUARD_GUARD (db,blk | all)
# SHIMON=1 run the target under LD_PRELOAD=libfdguard.so
# STRACE=0 1 = also wrap the target in strace (failed fd syscalls only)
# TRACE=0 1 = FDGUARD_TRACE_ALL (one line per open/close; big)
# MINE_INT=3 seconds between generate 1
# HAMMER_ARGS=... override the hammer thread mix
# STOP_ON_HIT=0 1 = end the run 60 s after the first NON-OWNER db/blk close
# EXTRA=... extra target args
source "$REPRO/common.sh"
NAME=$1; DUR=${2:-600}
OUT=$RUNS/$NAME
LOADS=${LOADS:-rpc,p2p,tor,mine}; WQ=${WQ:-4}; RT=${RT:-2}; GUARD=${GUARD:-all}
SHIMON=${SHIMON:-1}; STRACE=${STRACE:-0}; TRACE=${TRACE:-0}; MINE_INT=${MINE_INT:-3}
STOP_ON_HIT=${STOP_ON_HIT:-0}
HAMMER_ARGS=${HAMMER_ARGS:---normal 8 --impatient 8 --slowfire 2 --halfopen 3 --pipeline 3 --garbage 2 --slowread 2 --longpoll 4 --burst-size 16 --burst-every 2}
has() { case ",$LOADS," in *",$1,"*) return 0;; esac; return 1; }
mkdir -p "$OUT/logs"; rm -f "$OUT/STOP"
DD_T=$OUT/target; DD_H1=$OUT/helper1; DD_H2=$OUT/helper2
exec > >(tee -a "$OUT/run.txt") 2>&1
echo "=== repro run $NAME $(date -u +%FT%TZ) dur=${DUR}s loads=$LOADS wq=$WQ rt=$RT guard=$GUARD shim=$SHIMON strace=$STRACE"
echo "netns check (must be ENETUNREACH=101): $(python3 -c 'import socket;print(socket.socket().connect_ex(("1.1.1.1",80)))')"
echo "binary: $DRAGONXD build-id: $(readelf -n "$DRAGONXD" | awk '/Build ID/{print $3}')"
echo "dragonxd already running in this namespace? $(pgrep -a -x dragonxd | wc -l) (other agents' nodes live in other namespaces)"
PIDS=()
cleanup() {
echo "--- cleanup $(date -u +%T)"
touch "$OUT/STOP"
for p in "${PIDS[@]}"; do kill "$p" 2>/dev/null; done
stop_daemon "$DD_H2" "$RPC_H2"; stop_daemon "$DD_H1" "$RPC_H1"
if [ -n "$TPID" ] && kill -0 "$TPID" 2>/dev/null; then
timeout 20 "$CLI" -regtest -datadir="$DD_T" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T stop >/dev/null 2>&1
for _ in $(seq 1 60); do kill -0 "$TPID" 2>/dev/null || break; sleep 1; done
# abandoned longpoll getblocktemplate calls can hold every -rpcthreads worker, so RPC stop may never run:
# fall back to SIGTERM (same clean StartShutdown path), SIGKILL only as a last resort
kill -0 "$TPID" 2>/dev/null && { echo "RPC stop did not finish in 60 s -> SIGTERM"; kill -TERM "$TPID"; }
for _ in $(seq 1 180); do kill -0 "$TPID" 2>/dev/null || break; sleep 1; done
kill -0 "$TPID" 2>/dev/null && { echo "target did not stop -> SIGKILL"; kill -9 "$TPID"; }
fi
[ -n "$WPID" ] && wait "$WPID" 2>/dev/null; echo "target exit status: $?"
for p in "${PIDS[@]}"; do kill -9 "$p" 2>/dev/null; done
summarize
}
summarize() {
local DL=$DD_T/regtest/debug.log
echo "=== markers in target debug.log"
for m in "${MARKERS[@]}"; do printf ' [%6d] %s\n' "$(grep -c -- "$m" "$DL" 2>/dev/null)" "$m"; done
echo " db.log bytes: $(stat -c %s "$DD_T/regtest/db.log" 2>/dev/null || echo absent)"
grep -q 'Shutdown: done' "$DL" && echo " clean shutdown: yes" || echo " clean shutdown: NO"
if [ "$SHIMON" = 1 ]; then
echo "=== fdguard reports"; grep -o '=== FDGUARD #[0-9]* [A-Z-]*' "$OUT/fdguard.log" | awk '{print $4}' | sort | uniq -c
grep '^S ' "$OUT/fdguard.log" | tail -1
python3 "$REPRO/../shim/symbolize.py" --summary "$OUT/fdguard.log" > "$OUT/fdguard.summary.txt" 2>&1
head -60 "$OUT/fdguard.summary.txt"
fi
echo "=== BDB-page-in-blockfile detector"
python3 "$REPRO/tools/scan_bdb.py" "$DD_T"/regtest/blocks/blk*.dat "$DD_T"/regtest/blocks/rev*.dat 2>&1 | tail -15
echo "=== end $(date -u +%FT%TZ)"
}
trap cleanup EXIT
trap 'exit 130' INT TERM
echo "--- prep datadirs"
prep_golden "$DD_T"
prep_fresh "$DD_H1" "$RPC_H1"; prep_fresh "$DD_H2" "$RPC_H2"
cat > "$OUT/blocknotify.sh" <<EOF
#!/bin/sh
# pool-like blocknotify: a couple of dragonx-cli calls per block
$CLI -regtest -datadir=$DD_T -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T getblock "\$1" >/dev/null 2>&1
$CLI -regtest -datadir=$DD_T -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T getblocktemplate >/dev/null 2>&1
echo "\$(date +%s) \$1" >> $OUT/logs/blocknotify.log
EOF
chmod +x "$OUT/blocknotify.sh"
if has tor; then
nice -n 10 python3 "$REPRO/tools/faketor.py" --port $TORPORT > "$OUT/logs/faketor.txt" 2>&1 & PIDS+=($!)
TORARGS="-listenonion=1 -torcontrol=127.0.0.1:$TORPORT -debug=tor"
else
TORARGS="-listenonion=0"
fi
if has p2p || has p2pmin; then P2PARGS="-listen=1 -bind=127.0.0.1 -port=$P2P_T -allowlist=127.0.0.0/8 -maxinboundfromip=100"
else P2PARGS="-listen=0 -port=$P2P_T"; fi
TARGS=(-regtest -datadir="$DD_T" -testnode=1 -checkpoints=0 $P2PARGS -connect=127.0.0.1:1 -dnsseed=0 -dns=0
-asmap="$DD_T/asmap.dat" -rpcuser=$RPCUSER -rpcpassword=$RPCPASS -rpcport=$RPC_T
-txindex=1 -autoshield=1 -rpcworkqueue=$WQ -rpcthreads=$RT
"-blocknotify=$OUT/blocknotify.sh %s" $TORARGS -debug=net -debug=tls -printtoconsole=0 -daemon=0 $EXTRA)
echo "target args: ${TARGS[*]}" | sed "s/$RPCPASS/***/g"
ENVV=()
if [ "$SHIMON" = 1 ]; then
ENVV=(FDGUARD_LOG="$OUT/fdguard.log" FDGUARD_GUARD="$GUARD" FDGUARD_ABORT=0 FDGUARD_STATS_SEC=60 FDGUARD_MAXREPORTS=2000 LD_PRELOAD="$SHIM")
[ "$TRACE" = 1 ] && ENVV+=(FDGUARD_TRACE_ALL=1)
fi
WRAP=()
[ "$STRACE" = 1 ] && WRAP=(strace -f -tt -qq -e trace=close,openat,accept4,socket,dup2,dup3 -e status=failed -o "$OUT/strace.txt")
( cd "$DD_T" && exec env "${ENVV[@]}" nice -n 10 "${WRAP[@]}" "$DRAGONXD" "${TARGS[@]}" ) > "$OUT/logs/target_stdout.txt" 2>&1 &
WPID=$!
echo "--- target launched (wrapper pid $WPID); waiting for RPC"
waitrpc "$DD_T" "$RPC_T" 600 || { echo "target never answered RPC"; exit 1; }
TPID=$(cat "$DD_T/regtest/hushd.pid" 2>/dev/null); [ -z "$TPID" ] && TPID=$WPID
echo "target up pid=$TPID height=$(clin "$DD_T" "$RPC_T" getblockcount) $(date -u +%T)"
T0=$(date +%s)
if has p2p; then
start_helper "$DD_H1" "$P2P_H1" "$RPC_H1" "$OUT/logs/helper1.txt"
start_helper "$DD_H2" "$P2P_H2" "$RPC_H2" "$OUT/logs/helper2.txt"
waitrpc "$DD_H1" "$RPC_H1" 120; waitrpc "$DD_H2" "$RPC_H2" 120
nice -n 10 python3 "$REPRO/tools/tcpchurn.py" --port $P2P_T --duration "$DUR" --concurrency 8 > "$OUT/logs/tcpchurn.txt" 2>&1 & PIDS+=($!)
nice -n 10 bash "$REPRO/tools/rpcchurn.sh" "$OUT" "$DUR" > "$OUT/logs/rpcchurn.txt" 2>&1 & PIDS+=($!)
fi
if has p2pmin; then
# minimal P2P: ONE well-behaved helper peer, and an ordinary "disconnectnode" every DISC_INT s
# (the helper's -addnode re-dials the target). No raw TCP/TLS churn, no bans, no kills.
start_helper "$DD_H1" "$P2P_H1" "$RPC_H1" "$OUT/logs/helper1.txt"; waitrpc "$DD_H1" "$RPC_H1" 120
( k=0; while [ ! -e "$OUT/STOP" ]; do sleep "${DISC_INT:-5}"; for a in $(clin "$DD_T" "$RPC_T" getpeerinfo 2>/dev/null | python3 -c 'import json,sys; [print(p["addr"]) for p in json.load(sys.stdin)]' 2>/dev/null); do
clin "$DD_T" "$RPC_T" disconnectnode "$a" >/dev/null 2>&1 && { k=$((k+1)); echo "$(date -u +%T.%N) disconnectnode $a" >> "$OUT/logs/disconnects.log"; }; done
sleep 1; clin "$DD_H1" "$RPC_H1" addnode "127.0.0.1:$P2P_T" onetry >/dev/null 2>&1 # helper re-dials now (its -addnode retry is ~2 min)
echo "$k" > "$OUT/logs/disconnects.txt"; done ) & PIDS+=($!)
fi
if has rpc; then
MINEARG=""; has mine && MINEARG="--mine --mine-interval $MINE_INT"
nice -n 10 python3 "$REPRO/tools/hammer.py" --port $RPC_T --user $RPCUSER --password $RPCPASS --duration "$DUR" \
$HAMMER_ARGS $MINEARG --pid "$TPID" --logfile "$OUT/logs/hammer.log" > "$OUT/logs/hammer_stdout.txt" 2>&1 & PIDS+=($!)
elif has mine; then
( while [ ! -e "$OUT/STOP" ]; do clin "$DD_T" "$RPC_T" generate 1 >/dev/null 2>&1; sleep "$MINE_INT"; done ) & PIDS+=($!)
fi
# ---- poll
end=$(( T0 + DUR )); n=0; hit_at=0
while [ "$(date +%s)" -lt "$end" ] && [ ! -e "$OUT/STOP" ]; do
sleep 30; n=$((n+1))
alive=no; kill -0 "$TPID" 2>/dev/null && alive=yes
DL=$DD_T/regtest/debug.log
h=$(clin "$DD_T" "$RPC_T" getblockcount 2>/dev/null); c=$(clin "$DD_T" "$RPC_T" getconnectioncount 2>/dev/null)
rep=$(grep -c '^=== FDGUARD' "$OUT/fdguard.log" 2>/dev/null); noc=$(grep -c 'NON-OWNER-CLOSE' "$OUT/fdguard.log" 2>/dev/null)
ebf=$(grep -c 'EBADF-CLOSE' "$OUT/fdguard.log" 2>/dev/null); epl=$(grep -c '=== FDGUARD #[0-9]* EPOLL' "$OUT/fdguard.log" 2>/dev/null)
dbnoc=$(grep -A3 'NON-OWNER-CLOSE' "$OUT/fdguard.log" 2>/dev/null | grep -c 'class=\(db\|blk\)')
printf '[poll %3d %s +%4ds] alive=%s h=%s conns=%s | fdguard reports=%s nonowner=%s(db/blk %s) ebadf=%s epoll=%s | Epoll=%s SockCloseFail=%s EBADF=%s EPERM=%s SSL_write=%s seek=%s sel=%s | db.log=%s\n' \
$n "$(date -u +%T)" $(( $(date +%s) - T0 )) $alive "$h" "$c" "$rep" "$noc" "$dbnoc" "$ebf" "$epl" \
"$(grep -c 'libevent: Epoll' "$DL")" "$(grep -c 'Socket close failed' "$DL")" "$(grep -c 'Bad file descriptor' "$DL")" \
"$(grep -c 'Operation not permitted' "$DL")" "$(grep -c 'SSL_write' "$DL")" "$(grep -c 'Unable to seek' "$DL")" \
"$(grep -c 'socket select error' "$DL")" "$(stat -c %s "$DD_T/regtest/db.log" 2>/dev/null)" | tee -a "$OUT/poll.txt"
[ "$alive" = no ] && { echo "!!! target died"; break; }
if [ "$STOP_ON_HIT" = 1 ] && [ "${dbnoc:-0}" -gt 0 ]; then
[ "$hit_at" = 0 ] && hit_at=$(date +%s)
[ $(( $(date +%s) - hit_at )) -ge 60 ] && { echo "--- stop-on-hit"; break; }
fi
done
cat "$OUT/logs/tcpchurn.txt" "$OUT/logs/rpcchurn.txt" 2>/dev/null
grep -E 'TOTAL|totals' -A12 "$OUT/logs/hammer.log" 2>/dev/null | tail -14
qa/fd-guard/repro/tools/faketor.py
Executable
+98
View File
@@ -0,0 +1,98 @@
#!/usr/bin/env python3
"""
Fake Tor control port for the fd-bug harness.
dragonxd's torcontrol.cpp opens a libevent bufferevent to -torcontrol=<host:port>,
speaks the control protocol (PROTOCOLINFO / AUTHCHALLENGE / AUTHENTICATE / ADD_ONION),
and on EOF/ERROR reconnects with exponential backoff. We give it a server on a
loopback port that behaves badly in several ways so the reconnect/teardown loop
(bufferevent_free with BEV_OPT_CLOSE_ON_FREE, the event_base, the reconnect timer)
runs constantly. Each accepted connection picks a behaviour at random:
close : accept, then immediately close (EOF right away)
rst : accept, set SO_LINGER 0, close -> sends RST
stall : accept, read nothing, hold the socket open then drop it
garbage : send non-control-protocol bytes then close
halfok : answer PROTOCOLINFO plausibly, then drop before AUTHENTICATE completes
Loopback only. Run with --port. Ctrl-C / SIGTERM stops it.
"""
import argparse, random, socket, struct, threading, time, sys, signal
STOP = threading.Event()
def handle(conn, mode):
try:
conn.settimeout(5.0)
if mode == "close":
conn.close(); return
if mode == "rst":
conn.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER,
struct.pack("ii", 1, 0))
conn.close(); return
if mode == "stall":
# read whatever the controller sends but never reply; hold 2-4s
try:
conn.recv(4096)
except Exception:
pass
time.sleep(random.uniform(2.0, 4.0))
conn.close(); return
if mode == "garbage":
try:
conn.recv(4096)
except Exception:
pass
conn.sendall(b"\x00\xff\x01garbage not a tor reply\r\n" * 4)
conn.close(); return
if mode == "halfok":
# read the PROTOCOLINFO line, give a plausible 250 reply advertising
# NULL auth, then drop the connection before AUTHENTICATE finishes.
try:
conn.recv(4096)
except Exception:
pass
conn.sendall(b"250-PROTOCOLINFO 1\r\n"
b"250-AUTH METHODS=NULL\r\n"
b'250-VERSION Tor="0.4.7.13"\r\n'
b"250 OK\r\n")
time.sleep(random.uniform(0.2, 1.0))
conn.close(); return
except Exception:
try: conn.close()
except Exception: pass
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--port", type=int, required=True)
ap.add_argument("--host", default="127.0.0.1")
args = ap.parse_args()
modes = ["close", "rst", "stall", "garbage", "halfok"]
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind((args.host, args.port))
srv.listen(16)
srv.settimeout(1.0)
print(f"faketor: listening on {args.host}:{args.port}", flush=True)
def onsig(*_): STOP.set()
signal.signal(signal.SIGTERM, onsig)
signal.signal(signal.SIGINT, onsig)
n = 0
while not STOP.is_set():
try:
conn, _ = srv.accept()
except socket.timeout:
continue
except Exception:
break
mode = modes[n % len(modes)]
n += 1
threading.Thread(target=handle, args=(conn, mode), daemon=True).start()
srv.close()
print(f"faketor: stopped after {n} connections", flush=True)
if __name__ == "__main__":
main()
qa/fd-guard/repro/tools/hammer.py
Executable
+796
View File
@@ -0,0 +1,796 @@
#!/usr/bin/env python3
# hammer.py -- JSON-RPC robustness/load harness for a LOCAL regtest dragonxd.
#
# Scope: connects only to the --host/--port you pass (meant to be 127.0.0.1 and a
# regtest RPC port). Its job is to reproduce a known file-descriptor bug in the
# libevent-2.1.8 HTTP RPC server (src/httpserver.cpp) by exercising the same
# request patterns the production pool box sees: many concurrent clients, clients
# that disconnect mid-request/mid-response (the explorer indexer's 120 s client
# timeout), work-queue overflow, keep-alive pipelining, and malformed input.
# stdlib only.
#
# Correctness oracle: every request carries a unique JSON-RPC id; if a reply comes
# back with a different id on the same socket, that is cross-talk (the server wrote
# one client's reply onto another client's fd) and is logged + counted.
#
# See README.txt for the exact invocation.
import argparse
import base64
import collections
import json
import os
import random
import socket
import struct
import threading
import time
# ------------------------------------------------------------------ config/state
class C:
host = "127.0.0.1"
port = 19461
auth = ""
ctimeout = 15.0
CFG = C()
STOP = threading.Event()
SHARED = {"hashes": [], "lpid": None, "zaddrs": [], "rawblock": None}
SHARED_LOCK = threading.Lock()
_ID = [random.SystemRandom().randrange(1 << 40)]
_ID_LOCK = threading.Lock()
LOGLOCK = threading.Lock()
LOGF = [None]
def next_id():
with _ID_LOCK:
_ID[0] += 1
return _ID[0]
def log(msg):
line = "%s %s" % (time.strftime("%H:%M:%S"), msg)
with LOGLOCK:
print(line, flush=True)
if LOGF[0]:
LOGF[0].write(line + "\n"); LOGF[0].flush()
# ------------------------------------------------------------------ stats
KEYS = ["sent", "ok", "rpcerr", "wq500", "http401", "http4xx", "httpother",
"early_rst", "early_fin", "srvreset", "srveof", "ctimeout", "connfail",
"idmismatch", "malformed", "mined", "half_open", "half_srvclose"]
class Stats:
def __init__(self):
self.lk = threading.Lock()
self.total = collections.Counter()
self.win = collections.Counter()
self.anom = []
def inc(self, k, n=1):
with self.lk:
self.total[k] += n; self.win[k] += n
def anomaly(self, kind, detail):
with self.lk:
if len(self.anom) < 300:
self.anom.append((time.time(), kind, detail))
log("ANOMALY %s %s" % (kind, detail))
def swap(self):
with self.lk:
w = self.win; self.win = collections.Counter(); return w
ST = Stats()
# ------------------------------------------------------------------ socket helpers
def auth_header(u, p):
return "Basic " + base64.b64encode(("%s:%s" % (u, p)).encode()).decode()
def mkconn(timeout=None, rcvbuf=None):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
if rcvbuf:
s.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, rcvbuf)
s.settimeout(timeout if timeout is not None else CFG.ctimeout)
s.connect((CFG.host, CFG.port))
return s
def end_rst(s):
# linger 0 -> TCP RST on close (models a client that dies / resets abruptly)
try:
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
except OSError:
pass
try:
s.close()
except OSError:
pass
def end_fin(s):
try:
s.shutdown(socket.SHUT_RDWR)
except OSError:
pass
try:
s.close()
except OSError:
pass
def end_early(s, how=None):
how = how or random.choice(("rst", "fin"))
if how == "rst":
end_rst(s); ST.inc("early_rst")
else:
end_fin(s); ST.inc("early_fin")
def req_bytes(method, params, rid, keepalive=True, auth=None, uri="/",
httpmethod="POST", body=None, clen=None):
if body is None:
body = json.dumps({"jsonrpc": "1.0", "id": rid,
"method": method, "params": params}).encode()
head = [
"%s %s HTTP/1.1" % (httpmethod, uri),
"Host: %s:%d" % (CFG.host, CFG.port),
"Authorization: %s" % (auth if auth is not None else CFG.auth),
"Content-Type: application/json",
"Content-Length: %d" % (clen if clen is not None else len(body)),
"Connection: %s" % ("keep-alive" if keepalive else "close"),
"", "",
]
return "\r\n".join(head).encode() + body
class Reader:
"""Incremental HTTP/1.1 reply reader over one socket."""
def __init__(self, s):
self.s = s; self.buf = b""
def _fill(self):
d = self.s.recv(65536)
if not d:
raise EOFError()
self.buf += d
def read(self):
while b"\r\n\r\n" not in self.buf:
self._fill()
if len(self.buf) > (1 << 20):
raise ValueError("oversize header")
head, self.buf = self.buf.split(b"\r\n\r\n", 1)
lines = head.split(b"\r\n")
p = lines[0].split(b" ", 2)
if len(p) < 2 or not p[0].startswith(b"HTTP/1."):
raise ValueError("bad status line %r" % lines[0][:80])
status = int(p[1])
hdr = {}
for ln in lines[1:]:
if b":" in ln:
k, v = ln.split(b":", 1)
hdr[k.strip().lower()] = v.strip()
if hdr.get(b"transfer-encoding", b"").lower() == b"chunked":
body = b""
while True:
while b"\r\n" not in self.buf:
self._fill()
ln, self.buf = self.buf.split(b"\r\n", 1)
n = int(ln.split(b";")[0], 16)
while len(self.buf) < n + 2:
self._fill()
body += self.buf[:n]; self.buf = self.buf[n + 2:]
if n == 0:
break
elif b"content-length" in hdr:
n = int(hdr[b"content-length"])
while len(self.buf) < n:
self._fill()
body, self.buf = self.buf[:n], self.buf[n:]
else:
try:
while True:
self._fill()
except EOFError:
pass
body, self.buf = self.buf, b""
return status, hdr, body
def account(status, body, rid, method, expect401=False):
if status == 401:
ST.inc("http401")
if not expect401:
ST.anomaly("unexpected401", (method,))
return None
if status == 500 and b"Work queue depth exceeded" in body:
ST.inc("wq500"); return None
j = None
if body[:1] in (b"{", b"["):
try:
j = json.loads(body)
except ValueError:
j = None
if j is None:
if 400 <= status < 500:
ST.inc("http4xx")
else:
ST.inc("httpother")
if status == 200:
ST.anomaly("non-json-200", (method, body[:120].decode("latin1")))
return None
if isinstance(j, dict) and rid is not None and j.get("id") != rid:
ST.inc("idmismatch")
ST.anomaly("idmismatch", {"method": method, "sent": rid, "got": j.get("id"),
"body": body[:200].decode("latin1")})
if isinstance(j, dict) and j.get("error"):
ST.inc("rpcerr")
else:
ST.inc("ok")
return j
def call(s, rd, method, params, keepalive=True, expect401=False, auth=None):
rid = next_id()
try:
s.sendall(req_bytes(method, params, rid, keepalive=keepalive, auth=auth))
ST.inc("sent")
status, hdr, body = rd.read()
except socket.timeout:
ST.inc("ctimeout"); return None, False
except (ConnectionResetError, BrokenPipeError):
ST.inc("srvreset"); return None, False
except EOFError:
ST.inc("srveof"); return None, False
except ValueError as e:
ST.inc("malformed"); ST.anomaly("malformed", (method, str(e))); return None, False
except OSError:
ST.inc("srvreset"); return None, False
j = account(status, body, rid, method, expect401)
alive = keepalive and hdr.get(b"connection", b"").lower() != b"close"
return j, alive
# ------------------------------------------------------------------ menus
def a_hash():
with SHARED_LOCK:
return random.choice(SHARED["hashes"]) if SHARED["hashes"] else None
def a_zaddr():
with SHARED_LOCK:
return random.choice(SHARED["zaddrs"]) if SHARED["zaddrs"] else None
def normal_menu():
# Mostly fast calls (sub-ms) so a well-behaved client interleaves with mining.
# Heavy wallet calls (z_gettotalbalance ~2.3s on the golden wallet) appear but
# are rare, so they don't permanently peg the small worker pool on their own.
fast = [("getinfo", []), ("getblockcount", []), ("getbestblockhash", []),
("getrawmempool", []), ("getmininginfo", []), ("getdifficulty", []),
("getblockchaininfo", []), ("getpeerinfo", []),
("getblocktemplate", []), ("getnetworkinfo", [])]
h = a_hash()
if h:
fast.append(("getblock", [h, 2]))
fast.append(("getblockheader", [h]))
# "slow" here is only mildly-larger fast calls; the genuinely heavy wallet calls
# live in the paced slow-fire class so a normal dashboard poller never pegs the
# worker pool.
slow = [("coinsupply", []), ("listtransactions", ["*", 10]),
("listunspent", [0]), ("getblock", [a_hash() or "", 2])]
return random.choice(slow) if random.random() < 0.1 else random.choice(fast)
def medium_menu():
# Fast to compute (~4 ms on this node) but a non-trivial reply to write, so they
# are good targets for a mid-response disconnect. The fd-reuse race fires on the
# disconnect, not on call duration, so firing these at a HIGH rate drives the
# race hard while the worker frees in milliseconds (mining keeps its slots).
m = [("listtransactions", ["*", 10000]), ("getrawmempool", []),
("getrawmempool", [True]), ("getblockchaininfo", []),
("getblocktemplate", []), ("getchaintips", []), ("getinfo", []),
("coinsupply", []), ("listunspent", [0])]
h = a_hash()
if h:
m.append(("getblock", [h, 2]))
m.append(("getblock", [h, 1]))
return random.choice(m)
def heavy_menu():
# Genuinely slow calls (z_gettotalbalance ~1.7 s, z_listunspent ~5.7 s on the
# golden wallet). Fired only by the small, PACED slow-fire class so a worker
# keeps running a multi-second call long after the client disconnected (the
# production indexer/coinsupply scenario that widens the fd-reuse window),
# without the fire-and-disconnect backlog pinning the whole pool.
m = [("z_gettotalbalance", [0]), ("z_listunspent", [0])]
z = a_zaddr()
if z:
m.append(("z_listreceivedbyaddress", [z, 0]))
return random.choice(m)
# ------------------------------------------------------------------ clients
def client_normal(cid):
while not STOP.is_set():
keepalive = random.random() < 0.7
try:
s = mkconn()
except OSError:
ST.inc("connfail"); time.sleep(0.05); continue
rd = Reader(s)
n = random.randint(1, 8) if keepalive else 1
try:
for _ in range(n):
if STOP.is_set():
break
method, params = normal_menu()
j, alive = call(s, rd, method, params, keepalive=keepalive)
if not alive:
break
finally:
(end_fin if random.random() < 0.8 else end_rst)(s)
time.sleep(random.uniform(0.01, 0.15))
def client_impatient(cid):
# High-rate fast-disconnect churn: send a (medium) request, then disconnect
# after 1-500 ms at a random phase. This is the main driver of the fd-reuse
# race -- a disconnect concurrent with the server writing the reply / re-arming
# the socket -- and the worker frees in ms so mining is unaffected.
while not STOP.is_set():
method, params = medium_menu()
rid = next_id()
body = json.dumps({"jsonrpc": "1.0", "id": rid, "method": method,
"params": params}).encode()
phase = random.choice(("mid_request", "after_send", "mid_response"))
try:
s = mkconn(timeout=2.0)
except OSError:
ST.inc("connfail"); continue
try:
if phase == "mid_request":
full = req_bytes(method, params, rid, keepalive=False, body=body)
cut = random.randint(len(full) - len(body), len(full) - 1)
s.sendall(full[:cut])
time.sleep(random.uniform(0.001, 0.5))
end_early(s); continue
s.sendall(req_bytes(method, params, rid, keepalive=False, body=body))
ST.inc("sent")
time.sleep(random.uniform(0.001, 0.5))
if phase == "mid_response":
try:
s.recv(random.randint(0, 64)) # read a few bytes, then bail
except OSError:
pass
end_early(s)
except (OSError, socket.timeout):
ST.inc("srvreset")
end_rst(s)
time.sleep(random.uniform(0.05, 0.4))
def client_slowfire(cid, pace):
# Paced indexer/coinsupply client: fire ONE genuinely-slow request, disconnect
# after 1-500 ms (so the worker runs the multi-second call with no client left
# == the production fd-reuse window), then wait `pace` s before the next so a
# fire-and-disconnect backlog cannot pin the whole worker pool. Phase: cut the
# request mid-body, or send it whole and abort while the worker is busy, or read
# a few bytes of the reply then abort.
while not STOP.is_set():
method, params = heavy_menu()
rid = next_id()
body = json.dumps({"jsonrpc": "1.0", "id": rid, "method": method,
"params": params}).encode()
phase = random.choice(("mid_request", "after_send", "mid_response"))
try:
s = mkconn(timeout=2.0)
except OSError:
ST.inc("connfail"); time.sleep(pace); continue
try:
if phase == "mid_request":
full = req_bytes(method, params, rid, keepalive=False, body=body)
cut = random.randint(len(full) - len(body), len(full) - 1)
s.sendall(full[:cut])
time.sleep(random.uniform(0.001, 0.5)); end_early(s)
else:
s.sendall(req_bytes(method, params, rid, keepalive=False, body=body))
ST.inc("sent")
time.sleep(random.uniform(0.001, 0.5))
if phase == "mid_response":
try:
s.recv(random.randint(0, 64))
except OSError:
pass
end_early(s)
except (OSError, socket.timeout):
ST.inc("srvreset"); end_rst(s)
time.sleep(random.uniform(pace * 0.5, pace * 1.5))
def client_half_open(cid):
# connect, send nothing / partial headers, trickle, then vanish
while not STOP.is_set():
try:
s = mkconn(timeout=1.0)
except OSError:
ST.inc("connfail"); time.sleep(0.1); continue
ST.inc("half_open")
style = random.choice(("silent", "partial_headers", "full_headers_no_body"))
try:
if style == "partial_headers":
s.sendall(b"POST / HTTP/1.1\r\nHost: x\r\n")
elif style == "full_headers_no_body":
s.sendall(b"POST / HTTP/1.1\r\nHost: x\r\nAuthorization: " +
CFG.auth.encode() + b"\r\nContent-Length: 1000\r\n\r\n")
deadline = time.monotonic() + random.uniform(0.1, 20.0)
while time.monotonic() < deadline and not STOP.is_set():
time.sleep(random.uniform(0.2, 1.5))
try:
s.sendall(b" ") # one byte, keep it half-alive
except OSError:
ST.inc("half_srvclose"); break
try:
s.settimeout(0.01)
if s.recv(1) == b"":
ST.inc("half_srvclose"); break
except socket.timeout:
pass
except OSError:
ST.inc("half_srvclose"); break
finally:
end_early(s)
def client_burst(cid, size, every):
while not STOP.is_set():
batch = []
for _ in range(size):
try:
s = mkconn(timeout=3.0)
except OSError:
ST.inc("connfail"); continue
batch.append(s)
for s in batch:
try:
method, params = medium_menu()
s.sendall(req_bytes(method, params, next_id(), keepalive=False))
ST.inc("sent")
except OSError:
pass
# most close immediately (before the 500/reply is written); a few read it
keep = random.sample(batch, max(0, len(batch) // 4)) if batch else []
for s in batch:
if s in keep:
continue
end_early(s)
for s in keep:
try:
rd = Reader(s); st, hd, bd = rd.read(); account(st, bd, None, "burst")
except Exception:
ST.inc("srvreset")
finally:
end_fin(s)
for _ in range(int(every * 10)):
if STOP.is_set():
break
time.sleep(0.1)
def client_pipeline(cid):
while not STOP.is_set():
try:
s = mkconn()
except OSError:
ST.inc("connfail"); time.sleep(0.05); continue
n = random.randint(2, 8)
ids = []
blob = b""
for _ in range(n):
rid = next_id(); ids.append(rid)
method, params = normal_menu()
blob += req_bytes(method, params, rid, keepalive=True)
try:
s.sendall(blob); ST.inc("sent", n)
rd = Reader(s)
toread = random.randint(0, n) # sometimes leave replies unread, then close
for k in range(toread):
st, hd, bd = rd.read()
account(st, bd, ids[k], "pipeline")
except (OSError, EOFError, ValueError):
ST.inc("srvreset")
finally:
end_early(s)
time.sleep(random.uniform(0, 0.03))
def client_garbage(cid):
probes = [
("submitblock", lambda: ["ab" * random.randint(1, 4000)]),
("submitblock", lambda: [os.urandom(random.randint(1, 200)).hex()]),
("submitblock", lambda: ["zzzz"]),
("submitblock", lambda: [""]),
]
while not STOP.is_set():
pick = random.random()
try:
s = mkconn(timeout=5.0)
except OSError:
ST.inc("connfail"); continue
try:
if pick < 0.5:
method, mk = random.choice(probes)
j, _ = call(s, Reader(s), method, mk(), keepalive=False)
elif pick < 0.65:
# malformed JSON body
s.sendall(req_bytes("x", [], next_id(), keepalive=False,
body=b'{"jsonrpc":"1.0","method":'))
ST.inc("sent")
try:
account(*((lambda r: (r[0], r[2]))(Reader(s).read())), rid=None, method="badjson")
except Exception:
ST.inc("srvreset")
elif pick < 0.8:
# unknown method / wrong URI
s.sendall(req_bytes("getinfo", [], next_id(), keepalive=False, uri="/nope"))
ST.inc("sent")
try:
st, hd, bd = Reader(s).read(); account(st, bd, None, "wronguri")
except Exception:
ST.inc("srvreset")
else:
# Content-Length larger than the body we send, then bail
body = b'{"id":1}'
s.sendall(req_bytes("getinfo", [], 1, keepalive=False, body=body, clen=len(body) + 5000))
ST.inc("sent")
time.sleep(random.uniform(0.001, 0.3))
end_early(s)
continue
except (OSError, socket.timeout):
ST.inc("srvreset")
finally:
end_early(s)
time.sleep(random.uniform(0.02, 0.2))
def client_slowread(cid):
# ask for a big reply with a tiny recv buffer and do not read it, stalling the
# server write side, then close abruptly
while not STOP.is_set():
try:
s = mkconn(timeout=3.0, rcvbuf=512)
except OSError:
ST.inc("connfail"); continue
try:
s.sendall(req_bytes("listtransactions", ["*", 10000], next_id(), keepalive=False))
ST.inc("sent")
time.sleep(random.uniform(0.05, 1.0))
except OSError:
ST.inc("srvreset")
finally:
end_early(s)
time.sleep(random.uniform(0, 0.05))
def _stale_lpid():
# A longpollid whose hash != the current tip makes getblocktemplate return at
# once (the server's wait loop condition is immediately false), so it exercises
# the long-poll request path (lpval parsing, cs_main leave/enter) and the
# mid-response disconnect WITHOUT parking a worker. On a tiny -rpcthreads pool a
# genuinely parking long-poll + RPC-driven mining self-deadlocks (both workers
# wait for a block that only a third worker could mine); that is a pool-size
# artifact, not the fd bug, so the harness avoids it by default.
with SHARED_LOCK:
lpid = SHARED["lpid"]
if lpid and len(lpid) > 64:
h = lpid[:64]
flip = "0" if h[0] != "0" else "1"
return flip + h[1:] + lpid[64:]
return ("0" * 64) + "0"
def client_longpoll(cid):
while not STOP.is_set():
try:
s = mkconn(timeout=10.0)
except OSError:
ST.inc("connfail"); time.sleep(0.2); continue
rd = Reader(s)
params = [{"longpollid": _stale_lpid()}]
rid = next_id()
try:
s.sendall(req_bytes("getblocktemplate", params, rid, keepalive=False))
ST.inc("sent")
if random.random() < 0.5:
# impatient pool worker: abandon the request after 1-500 ms
time.sleep(random.uniform(0.001, 0.5))
end_early(s); continue
st, hd, bd = rd.read()
j = account(st, bd, rid, "getblocktemplate(lp)")
if j and isinstance(j, dict) and j.get("result"):
with SHARED_LOCK:
SHARED["lpid"] = j["result"].get("longpollid")
except (OSError, socket.timeout, EOFError, ValueError):
ST.inc("srvreset")
finally:
end_fin(s)
time.sleep(random.uniform(0.02, 0.2))
def miner(interval):
# Mine a block every `interval` seconds. Retry through work-queue rejections so
# blocks still land under heavy load (the production box mines steadily too).
while not STOP.is_set():
got = False
deadline = time.monotonic() + interval + 30
while not got and not STOP.is_set() and time.monotonic() < deadline:
try:
s = mkconn(timeout=60.0)
j, _ = call(s, Reader(s), "generate", [1], keepalive=False)
end_fin(s)
if j and not j.get("error") and j.get("result"):
ST.inc("mined"); got = True
h = j["result"][0] if isinstance(j["result"], list) else None
if h:
with SHARED_LOCK:
SHARED["hashes"].append(h)
SHARED["hashes"] = SHARED["hashes"][-200:]
else:
time.sleep(0.2) # likely work-queue 500; back off and retry
except Exception:
time.sleep(0.2)
for _ in range(int(interval * 10)):
if STOP.is_set():
break
time.sleep(0.1)
# ------------------------------------------------------------------ bootstrap shared state
def bootstrap():
try:
s = mkconn(timeout=30.0); rd = Reader(s)
j, _ = call(s, rd, "getbestblockhash", [], keepalive=True)
if j and j.get("result"):
with SHARED_LOCK:
SHARED["hashes"].append(j["result"])
j, _ = call(s, rd, "getblockcount", [], keepalive=True)
cnt = j.get("result") if j else None
if isinstance(cnt, int):
for hh in range(max(0, cnt - 30), cnt + 1):
jj, _ = call(s, rd, "getblockhash", [hh], keepalive=True)
if jj and jj.get("result"):
with SHARED_LOCK:
SHARED["hashes"].append(jj["result"])
j, _ = call(s, rd, "listaddresses", [], keepalive=True)
j, _ = call(s, rd, "z_listaddresses", [], keepalive=True)
if j and isinstance(j.get("result"), list):
with SHARED_LOCK:
SHARED["zaddrs"] = j["result"][:16]
j, _ = call(s, rd, "getblocktemplate", [{}], keepalive=True)
if j and isinstance(j.get("result"), dict):
with SHARED_LOCK:
SHARED["lpid"] = j["result"].get("longpollid")
end_fin(s)
log("bootstrap: %d hashes, %d zaddrs, lpid=%s" %
(len(SHARED["hashes"]), len(SHARED["zaddrs"]), bool(SHARED["lpid"])))
except Exception as e:
log("bootstrap failed: %s" % e)
# ------------------------------------------------------------------ fd accounting
def count_fds(pid):
if not pid:
return (None, None, None)
base = "/proc/%d/fd" % pid
try:
fds = os.listdir(base)
except OSError:
return (None, None, None)
sock = files = 0
for fd in fds:
try:
tgt = os.readlink(os.path.join(base, fd))
except OSError:
continue
if tgt.startswith("socket:"):
sock += 1
elif tgt.startswith("/"):
files += 1
return (len(fds), sock, files)
def pid_alive(pid):
if not pid:
return True
try:
os.kill(pid, 0); return True
except OSError:
return False
# ------------------------------------------------------------------ main
def reporter(pid):
t0 = time.monotonic()
while not STOP.is_set():
for _ in range(10):
if STOP.is_set():
break
time.sleep(0.1)
w = ST.swap()
tot, sk, fl = count_fds(pid)
fdstr = ""
if tot is not None:
fdstr = " | fds=%d sock=%d file=%d" % (tot, sk, fl)
log("t=%4ds ok=%d rpcerr=%d wq500=%d 4xx=%d other=%d | "
"earlyRST=%d earlyFIN=%d srvRST=%d srvEOF=%d ctmo=%d connfail=%d | "
"idmis=%d malf=%d | mined=%d halfopen=%d%s" % (
int(time.monotonic() - t0),
w["ok"], w["rpcerr"], w["wq500"], w["http4xx"], w["httpother"],
w["early_rst"], w["early_fin"], w["srvreset"], w["srveof"],
w["ctimeout"], w["connfail"], w["idmismatch"], w["malformed"],
w["mined"], w["half_open"], fdstr))
if pid and not pid_alive(pid):
log("!!! NODE PROCESS %d IS GONE -- stopping harness" % pid)
STOP.set()
break
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, default=19461)
ap.add_argument("--user", default="ab")
ap.add_argument("--password", default="abpass")
ap.add_argument("--duration", type=int, default=300)
ap.add_argument("--ctimeout", type=float, default=15.0)
ap.add_argument("--normal", type=int, default=24)
ap.add_argument("--impatient", type=int, default=10)
ap.add_argument("--slowfire", type=int, default=3,
help="paced clients that fire a genuinely-slow call then disconnect")
ap.add_argument("--slowfire-pace", type=float, default=2.0)
ap.add_argument("--halfopen", type=int, default=6)
ap.add_argument("--pipeline", type=int, default=6)
ap.add_argument("--garbage", type=int, default=4)
ap.add_argument("--slowread", type=int, default=2)
ap.add_argument("--longpoll", type=int, default=6)
# burst-size must exceed (rpcthreads + rpcworkqueue) to overflow the queue even
# when calls are fast; with -rpcthreads=8 -rpcworkqueue=4 that threshold is 12.
ap.add_argument("--burst-threads", type=int, default=1)
ap.add_argument("--burst-size", type=int, default=24)
ap.add_argument("--burst-every", type=float, default=1.5)
ap.add_argument("--mine", action="store_true", help="run a miner thread")
ap.add_argument("--mine-interval", type=float, default=4.0)
ap.add_argument("--pid", type=int, default=0, help="node pid, for /proc fd + liveness")
ap.add_argument("--logfile", default="")
args = ap.parse_args()
CFG.host = args.host; CFG.port = args.port
CFG.auth = auth_header(args.user, args.password); CFG.ctimeout = args.ctimeout
if args.logfile:
LOGF[0] = open(args.logfile, "a")
log("hammer start host=%s port=%d duration=%ds pid=%s" %
(CFG.host, CFG.port, args.duration, args.pid or "-"))
bootstrap()
threads = []
def spawn(fn, n, *a):
for i in range(n):
t = threading.Thread(target=fn, args=(i,) + a, daemon=True)
t.start(); threads.append(t)
spawn(client_normal, args.normal)
spawn(client_impatient, args.impatient)
spawn(client_slowfire, args.slowfire, args.slowfire_pace)
spawn(client_half_open, args.halfopen)
spawn(client_pipeline, args.pipeline)
spawn(client_garbage, args.garbage)
spawn(client_slowread, args.slowread)
spawn(client_longpoll, args.longpoll)
for i in range(args.burst_threads):
t = threading.Thread(target=client_burst, args=(i, args.burst_size, args.burst_every), daemon=True)
t.start(); threads.append(t)
if args.mine:
t = threading.Thread(target=miner, args=(args.mine_interval,), daemon=True)
t.start(); threads.append(t)
rt = threading.Thread(target=reporter, args=(args.pid,), daemon=True)
rt.start()
end = time.monotonic() + args.duration
try:
while time.monotonic() < end and not STOP.is_set():
time.sleep(0.2)
except KeyboardInterrupt:
pass
STOP.set()
time.sleep(1.0)
log("=== TOTALS ===")
for k in KEYS:
if ST.total[k]:
log(" %-14s %d" % (k, ST.total[k]))
log("anomalies: %d" % len(ST.anom))
for t, kind, detail in ST.anom[:40]:
log(" %s %s %s" % (time.strftime("%H:%M:%S", time.localtime(t)), kind, detail))
if args.pid:
log("node alive at end: %s" % pid_alive(args.pid))
if __name__ == "__main__":
main()
qa/fd-guard/repro/tools/rpcchurn.sh
Executable
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
# RPC-driven P2P peer churn on the target (same mix as fdbug/p2p/rpcchurn.sh):
# disconnectnode / addnode onetry / setban+clearbanned / kill -9 + restart of helper2.
# usage: rpcchurn.sh <run dir> <seconds>
source "$REPRO/common.sh"
OUT=$1; DUR=${2:-300}
DD_T=$OUT/target; DD_H1=$OUT/helper1; DD_H2=$OUT/helper2
end=$(( $(date +%s) + DUR )); i=0; restarts=0
while [ "$(date +%s)" -lt "$end" ] && [ ! -e "$OUT/STOP" ]; do
i=$((i+1))
case $(( i % 5 )) in
0) clin "$DD_T" "$RPC_T" disconnectnode "127.0.0.1:$P2P_H1" >/dev/null 2>&1 ;;
1) clin "$DD_T" "$RPC_T" addnode "127.0.0.1:$P2P_H1" onetry >/dev/null 2>&1 ;;
2) clin "$DD_T" "$RPC_T" setban "127.0.0.1" add 1 >/dev/null 2>&1; sleep 0.3
clin "$DD_T" "$RPC_T" clearbanned >/dev/null 2>&1
clin "$DD_H2" "$RPC_H2" addnode "127.0.0.1:$P2P_T" onetry >/dev/null 2>&1 ;;
3) pid=$(cat "$DD_H2/regtest/hushd.pid" 2>/dev/null)
if [ -n "$pid" ] && kill -0 "$pid" 2>/dev/null; then
kill -9 "$pid" 2>/dev/null
for _ in $(seq 1 20); do kill -0 "$pid" 2>/dev/null || break; sleep 0.5; done
start_helper "$DD_H2" "$P2P_H2" "$RPC_H2" "$OUT/logs/helper2.txt"
waitrpc "$DD_H2" "$RPC_H2" 120 >/dev/null 2>&1
restarts=$((restarts+1))
fi ;;
4) clin "$DD_T" "$RPC_T" addnode "127.0.0.1:$P2P_H2" onetry >/dev/null 2>&1 ;;
esac
sleep 0.7
done
echo "rpcchurn: $i cycles, $restarts helper2 restarts"
qa/fd-guard/repro/tools/scan_bdb.py
Executable
+113
View File
@@ -0,0 +1,113 @@
#!/usr/bin/env python3
"""
scan_bdb.py -- find Berkeley DB pages written into files that must not contain them
(blocks/blk*.dat, blocks/rev*.dat). Stand-in for the production incident tool.
BDB writes page P of a database at file offset P*pagesize through pwrite(fd, ...). If
fd has been closed and its number reused by a block file, the page lands in the block
file at exactly P*pagesize. So at every pagesize-aligned offset we decode the PAGE
header (db_page.h, all little-endian on x86):
0 u32 lsn.file 4 u32 lsn.offset 8 u32 pgno 12 u32 prev_pgno
16 u32 next_pgno 20 u16 entries 22 u16 hf_offset
24 u8 level 25 u8 type
and a metadata page (pgno 0 / any meta type) also carries magic at 12 and pagesize at 20.
A hit needs ALL of:
* pgno == offset // pagesize (the self-locating property)
* type is a real BDB page type (1..16 except 0)
* level consistent with type (overflow/meta/hash: 0; btree leaf: 1; internal: 2..32)
* hf_offset <= pagesize; entries <= pagesize/2
* for overflow (7): entries (refcount) >= 1 and 0 < hf_offset (data length) <= pagesize-26
Strength: "STRONG" if lsn == (0,1) (pages rewritten by lsn_reset -- the production
signature) or a meta page with a valid magic, else "weak".
Also reports any aligned BDB meta magic (btree 0x053162, hash 0x061561, queue 0x042253,
heap 0x074582) even when pgno does not match.
usage: scan_bdb.py [--pagesize 4096] [--all-offsets] files...
exit 0 = clean, 2 = hits.
"""
import argparse, struct, sys
TYPES = {1: "duplicate", 2: "hash_unsorted", 3: "ibtree", 4: "irecno", 5: "lbtree", 6: "lrecno",
7: "overflow", 8: "hashmeta", 9: "btreemeta", 10: "qammeta", 11: "qamdata", 12: "ldup",
13: "hash", 14: "heapmeta", 15: "heap", 16: "iheap"}
MAGIC = {0x053162: "btree", 0x061561: "hash", 0x042253: "queue", 0x074582: "heap"}
META = {8, 9, 10, 14}
def level_ok(t, lvl):
if t in (5, 6, 12): # leaf pages
return lvl == 1
if t in (3, 4): # internal pages
return 2 <= lvl <= 32
return lvl == 0 # overflow / meta / hash / queue / heap
def check(buf, off, ps):
if len(buf) < 26:
return None
lf, lo, pgno, prv, nxt, ent, hfo, lvl, typ = struct.unpack_from("<IIIIIHHBB", buf, 0)
if typ not in TYPES or pgno != off // ps or not level_ok(typ, lvl):
return None
if typ in META:
magic, ver, psz = struct.unpack_from("<III", buf, 12)
if magic not in MAGIC:
return None
return ("STRONG", f"{TYPES[typ]} magic={MAGIC[magic]} ver={ver} pagesize={psz} lsn=({lf},{lo})")
if hfo > ps or ent > ps // 2:
return None
if typ == 7 and not (ent >= 1 and 0 < hfo <= ps - 26):
return None
strength = "STRONG" if (lf, lo) == (0, 1) else "weak"
return (strength, f"{TYPES[typ]} lsn=({lf},{lo}) prev={prv} next={nxt} entries={ent} hf_offset={hfo} level={lvl}")
def scan(path, ps):
hits, magic_only = [], []
with open(path, "rb") as f:
off = 0
while True:
buf = f.read(ps)
if not buf:
break
r = check(buf, off, ps)
if r:
hits.append((off, off // ps) + r)
elif len(buf) >= 16:
m = struct.unpack_from("<I", buf, 12)[0]
if m in MAGIC and buf[25] in META:
magic_only.append((off, MAGIC[m]))
off += len(buf)
return hits, magic_only, off
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--pagesize", type=int, default=4096)
ap.add_argument("--quiet", action="store_true")
ap.add_argument("files", nargs="+")
a = ap.parse_args()
total = strong = 0
for p in a.files:
try:
hits, mo, size = scan(p, a.pagesize)
except OSError as e:
print(f"{p}: ERROR {e}")
continue
s = sum(1 for h in hits if h[2] == "STRONG")
total += len(hits); strong += s
if hits or mo:
print(f"{p}: {len(hits)} BDB page(s) ({s} strong), {len(mo)} stray meta magic, size={size}")
if not a.quiet:
for h in hits[:20]:
print(f" off={h[0]} pgno={h[1]} {h[2]} {h[3]}")
for m in mo[:5]:
print(f" meta-magic off={m[0]} {m[1]}")
else:
print(f"{p}: clean ({size} bytes)")
print(f"TOTAL BDB pages found: {total} ({strong} strong)")
return 2 if total else 0
if __name__ == "__main__":
sys.exit(main())
qa/fd-guard/repro/tools/tally.py
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""tally.py <fdguard.log> -- attribute every fdguard report to (type, closer function, target class)
and, for EBADF/NON-OWNER reports, to the code that made the previous close of that fd number.
The point: show that the stray closes ORIGINATE in CNode code (net.cpp CloseSocketDisconnect /
~CNode closing a number they already closed) and that libevent / BDB / CreateNodeFromAcceptedSocket /
GetTransaction only appear as victims-turned-amplifiers downstream of that.
"""
import collections, re, sys
ORIGIN = ("_ZN5CNode21CloseSocketDisconnectEv", "_ZN5CNodeD1Ev", "_ZN5CNodeD2Ev", "_ZN11CNetCleanupD2Ev")
NAMES = {"_ZN5CNode21CloseSocketDisconnectEv": "CNode::CloseSocketDisconnect (net.cpp:572)",
"_ZN5CNodeD1Ev": "CNode::~CNode (net.cpp:2819)",
"_ZN11CNetCleanupD2Ev": "CNetCleanup::~CNetCleanup (net.cpp:2465, shutdown)",
"evhttp_connection_free": "libevent evhttp_connection_free (http.c:1244)",
"__os_closehandle": "BDB __os_closehandle",
"_Z28CreateNodeFromAcceptedSocketjbRK8CAddressS1_": "CreateNodeFromAcceptedSocket (net.cpp:1336)",
"_Z14GetTransactionRK7uint256R12CTransactionRS_b": "GetTransaction fclose (main.cpp)",
"epoll_nochangelist_add": "libevent epoll ADD", "epoll_nochangelist_del": "libevent epoll DEL/MOD"}
def fn(frame_line):
m = re.search(r"\+0x[0-9a-f]+ (\S+?)\+0x", frame_line)
return m.group(1) if m else "?"
def main(path):
reps, cur = [], None
for line in open(path, errors="replace"):
if line.startswith("=== FDGUARD #"):
cur = {"hdr": line.strip(), "lines": []}
reps.append(cur)
elif cur is not None:
if line.startswith("=== END"):
cur = None
else:
cur["lines"].append(line.rstrip("\n"))
tally = collections.Counter()
prev_tally = collections.Counter()
db_blk = []
for r in reps:
typ = r["hdr"].split()[3]
L = r["lines"]
cls = next((re.search(r"class=(\S+)", l).group(1) for l in L if " target=" in l and "class=" in l), "?")
try:
i = next(k for k, l in enumerate(L) if l.strip() == "closer stack:")
closer = fn(L[i + 1])
except StopIteration:
closer = "?"
prev = "-"
for k, l in enumerate(L):
if "previous successful close of this fd number" in l:
# first frame line after the target= line
for l2 in L[k + 1:k + 4]:
if l2.strip().startswith("#0"):
prev = fn(l2)
break
break
tally[(typ, NAMES.get(closer, closer), cls)] += 1
if typ in ("EBADF-CLOSE", "NON-OWNER-CLOSE"):
prev_tally[NAMES.get(prev, prev)] += 1
if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk"):
tgt = next((l.split("target=")[1].split()[0] for l in L if " target=" in l), "?")
db_blk.append((r["hdr"].split()[2], r["hdr"].split()[4], NAMES.get(closer, closer), tgt.rsplit("/", 2)[-2:]))
print(f"{len(reps)} reports in {path}")
print("\n count type class closer")
for (typ, closer, cls), n in tally.most_common():
print(f"{n:6d} {typ:17s} {cls:6s} {closer}")
print("\nEBADF + NON-OWNER reports by the code that made the PREVIOUS close of the same number:")
for k, n in prev_tally.most_common():
print(f"{n:6d} {k}")
o = sum(n for (typ, c, cls), n in tally.items() if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk") and c.startswith(("CNode", "CNetCleanup")))
a = sum(n for (typ, c, cls), n in tally.items() if typ == "NON-OWNER-CLOSE" and cls in ("db", "blk")) - o
print(f"\nNON-OWNER closes of wallet/BDB-log/block files: {o + a} (by CNode code directly: {o}; by an amplifier: {a})")
for d in db_blk[:40]:
print(" ", d[0], d[1], "closer=" + d[2], "target=" + "/".join(d[3]))
if __name__ == "__main__":
main(sys.argv[1])
qa/fd-guard/repro/tools/tcpchurn.py
Executable
+147
View File
@@ -0,0 +1,147 @@
#!/usr/bin/env python3
"""
Raw TCP / TLS client churn against the target node's P2P port.
dragonxd P2P is TLS 1.3 only (wolfSSL), self-signed, no cert verification. The
accept path is net.cpp CreateNodeFromAcceptedSocket -> tlsmanager.accept()
(wolfSSL_accept), and teardown is CloseSocketDisconnect / ~CNode
(wolfSSL_shutdown + wolfSSL_free + CloseSocket). This driver pokes the accept and
TLS-teardown/error paths (SSL_read/SSL_write at net.cpp ~1005 and tlsmanager.cpp
~660-680) by opening many short-lived connections that each misbehave:
rst : connect, then abort with RST (SO_LINGER 0) before any TLS bytes
fin : connect, then close cleanly (FIN) before any TLS bytes
garbage : connect, send non-ClientHello bytes, then drop
stall : connect, send a few bytes, hold mid-handshake, drop
tlsjunk : complete a real TLS 1.3 handshake (verify off), send junk, drop
tlshalf : start TLS, send a partial ClientHello record, drop
Loopback only, against a port this harness started. --duration seconds.
"""
import argparse, random, socket, ssl, struct, sys, time, os
def mk():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(3.0)
return s
def rst_close(s):
try:
s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack("ii", 1, 0))
except Exception:
pass
try: s.close()
except Exception: pass
def do_rst(host, port):
s = mk()
try:
s.connect((host, port)); rst_close(s)
except Exception:
rst_close(s)
def do_fin(host, port):
s = mk()
try:
s.connect((host, port)); s.close()
except Exception:
try: s.close()
except Exception: pass
def do_garbage(host, port):
s = mk()
try:
s.connect((host, port))
s.sendall(os.urandom(random.randint(16, 400)))
time.sleep(random.uniform(0.0, 0.3))
except Exception:
pass
rst_close(s)
def do_stall(host, port):
s = mk()
try:
s.connect((host, port))
# a couple of bytes that look like the start of a TLS record, then wait
s.sendall(b"\x16\x03\x01")
time.sleep(random.uniform(0.5, 2.0))
except Exception:
pass
rst_close(s)
def _ctx():
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
try:
ctx.minimum_version = ssl.TLSVersion.TLSv1_3
except Exception:
pass
return ctx
def do_tlsjunk(host, port):
raw = mk()
try:
raw.connect((host, port))
tls = _ctx().wrap_socket(raw, server_hostname=None)
# handshake completed; the node now expects a P2P version message.
# send junk inside the TLS channel, then tear down.
try:
tls.sendall(os.urandom(random.randint(8, 128)))
except Exception:
pass
time.sleep(random.uniform(0.0, 0.4))
try:
tls.close()
except Exception:
rst_close(raw)
except Exception:
rst_close(raw)
def do_tlshalf(host, port):
s = mk()
try:
s.connect((host, port))
# a well-formed TLS record header claiming a long handshake, but we send
# only part of it, then drop -> server sits in wolfSSL_accept WANT_READ.
body = os.urandom(40)
rec = b"\x16\x03\x03" + struct.pack(">H", 512) + body
s.sendall(rec)
time.sleep(random.uniform(0.3, 1.5))
except Exception:
pass
rst_close(s)
HANDLERS = [do_rst, do_fin, do_garbage, do_stall, do_tlsjunk, do_tlshalf]
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, required=True)
ap.add_argument("--duration", type=float, default=300.0)
ap.add_argument("--concurrency", type=int, default=6)
ap.add_argument("--rate", type=float, default=0.03, help="sleep between bursts")
args = ap.parse_args()
end = time.time() + args.duration
counts = {h.__name__: 0 for h in HANDLERS}
import threading
lock = threading.Lock()
def worker():
while time.time() < end:
h = random.choice(HANDLERS)
h(args.host, args.port)
with lock:
counts[h.__name__] += 1
time.sleep(random.uniform(0.0, args.rate))
ts = [threading.Thread(target=worker, daemon=True) for _ in range(args.concurrency)]
for t in ts: t.start()
for t in ts: t.join()
total = sum(counts.values())
print(f"tcpchurn: {total} connections in {args.duration:.0f}s -> {counts}", flush=True)
if __name__ == "__main__":
main()
qa/fd-guard/shim/build.sh
Executable
+15
View File
@@ -0,0 +1,15 @@
#!/bin/sh
# Build the fdguard LD_PRELOAD shim and its self-test program.
# ./build.sh -> libfdguard.so, fdguard_test
set -e
cd "$(dirname "$0")"
CC=${CC:-gcc}
$CC -O2 -g -fPIC -shared -Wall -Wextra -Wno-unused-parameter -fvisibility=default \
-o libfdguard.so.tmp fdguard.c -ldl -lpthread
# rename, never overwrite in place: a running process may have the old one mapped
mv -f libfdguard.so.tmp libfdguard.so
# the test program keeps its symbols (no -rdynamic on purpose: like dragonxd,
# owner detection must work from .symtab alone) and frame pointers for clarity
$CC -O1 -g -fno-omit-frame-pointer -fno-inline -Wall -o fdguard_test.tmp fdguard_test.c -lpthread
mv -f fdguard_test.tmp fdguard_test
echo "built: $(pwd)/libfdguard.so $(pwd)/fdguard_test"
qa/fd-guard/shim/fdguard.c
+1500
View File
File diff suppressed because it is too large Load Diff
qa/fd-guard/shim/fdguard_test.c
+206
View File
@@ -0,0 +1,206 @@
/*
* fdguard_test.c -- exercises libfdguard.so.
*
* The function names imitate the real owners inside dragonxd so the shim's
* built-in domain patterns apply unchanged:
* __os_openhandle / __os_closehandle Berkeley DB os layer (domain bdb)
* OpenDiskFile / FlushBlockFile block-file code (domain blk)
* ConnectSocketDirectly / CloseSocket P2P sockets (domain p2p)
* foreign_stale_close / foreign_* somebody who holds a stale number
*
* usage: fdguard_test <dir> <case> (see main() for the list)
*/
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/epoll.h>
#include <sys/socket.h>
#include <sys/syscall.h>
#include <time.h>
#include <unistd.h>
#define NI __attribute__((noinline))
static char D[512];
static char *P(const char *rel) { static char b[4][700]; static int i; i = (i + 1) & 3; snprintf(b[i], sizeof b[i], "%s/%s", D, rel); return b[i]; }
static void say(const char *m) { fprintf(stderr, "[test] %s\n", m); }
/* --- "Berkeley DB" --- */
NI int __os_openhandle(const char *path) { return open(path, O_RDWR | O_CREAT, 0600); }
NI int __os_closehandle(int fd) { return close(fd); }
/* --- "block files" --- */
NI FILE *OpenDiskFile(const char *path) { FILE *f = fopen(path, "rb+"); if (!f) f = fopen(path, "wb+"); return f; }
NI int FlushBlockFile(FILE *f) { fflush(f); return fclose(f); }
/* --- "P2P" --- */
NI int ConnectSocketDirectly(void) { return socket(AF_INET, SOCK_STREAM, 0); }
NI int CloseSocket(int *s) { int r = close(*s); if (r) fprintf(stderr, "[test] Socket close failed: %d. Error: %s\n", *s, strerror(errno)); return r; }
/* --- foreigners --- */
NI int foreign_stale_close(int fd) { return close(fd); }
NI int foreign_stale_fclose(FILE *f) { return fclose(f); }
NI int foreign_dup2(int fd) { int dn = open("/dev/null", O_RDWR); int r = dup2(dn, fd); close(dn); return r; }
NI int foreign_close_range(int fd) { return close_range((unsigned)fd, (unsigned)fd, 0); }
NI void foreign_closefrom(int fd) { closefrom(fd); }
/* --- plain code that opens and closes in the same function --- */
NI int copy_like(const char *src) { int fd = open(src, O_RDONLY); char b[16]; if (fd >= 0) { (void)!read(fd, b, sizeof b); close(fd); } return fd; }
NI int plain_open(const char *p) { return open(p, O_RDWR | O_CREAT, 0600); }
NI int plain_close(int fd) { return close(fd); }
static int fd_valid(int fd) { return fcntl(fd, F_GETFD) != -1; }
static int c_double(void)
{
int fd = plain_open(P("plain.txt"));
plain_close(fd);
int r = plain_close(fd); /* EBADF */
return r == -1 && errno == EBADF ? 0 : 1;
}
static int c_dbowner(void) { int fd = __os_openhandle(P("regtest/wallet.dat")); int r = __os_closehandle(fd); return r; }
static int c_dbforeign(void)
{
int fd = __os_openhandle(P("regtest/wallet.dat"));
foreign_stale_close(fd); /* HIT */
int r = __os_closehandle(fd); /* BDB's own close then fails: EBADF (unless deny) */
(void)r;
return 0;
}
static int c_deny(void)
{
int fd = __os_openhandle(P("regtest/wallet.dat"));
foreign_stale_close(fd); /* HIT + DENIED */
if (!fd_valid(fd)) { say("deny FAILED: fd closed"); return 1; }
if (write(fd, "x", 1) != 1) { say("deny FAILED: write"); return 1; }
return __os_closehandle(fd);
}
static int c_blkowner(void) { FILE *f = OpenDiskFile(P("regtest/blocks/blk00000.dat")); fputs("blk", f); return FlushBlockFile(f); }
static int c_blkraw(void)
{
FILE *f = OpenDiskFile(P("regtest/blocks/blk00001.dat"));
foreign_stale_close(fileno(f)); /* HIT: raw close under a live FILE */
FlushBlockFile(f); /* EBADF from fclose */
return 0;
}
static int c_stalefile(void)
{
FILE *f = OpenDiskFile(P("regtest/blocks/blk00002.dat"));
int n = fileno(f);
foreign_stale_close(n); /* HIT #1 */
int w = __os_openhandle(P("regtest/wallet.dat"));
if (w != n) fprintf(stderr, "[test] note: wallet got fd %d, FILE had %d\n", w, n);
FlushBlockFile(f); /* HIT #2: fclose of a stale FILE whose number is now wallet.dat */
if (w == n) __os_closehandle(w); /* EBADF: BDB lost its fd */
return 0;
}
static int c_reuse(void)
{
int s = ConnectSocketDirectly();
int keep = s;
CloseSocket(&s); /* legit */
int w = __os_openhandle(P("regtest/wallet.dat")); /* reuses the number */
if (w != keep) fprintf(stderr, "[test] note: wallet got fd %d, socket was %d\n", w, keep);
s = keep;
CloseSocket(&s); /* HIT: stale socket number now = wallet.dat */
__os_closehandle(w); /* EBADF */
return 0;
}
static int c_dup2(void) { int w = __os_openhandle(P("regtest/wallet.dat")); foreign_dup2(w); __os_closehandle(w); return 0; }
static int c_range(void)
{
int w = __os_openhandle(P("regtest/wallet.dat"));
foreign_close_range(w); /* HIT */
w = __os_openhandle(P("regtest/database/log.0000000001"));
int hi = fcntl(w, F_DUPFD, 600); /* move it high so closefrom only hits it */
close(w);
foreign_closefrom(hi); /* HIT (domain mismatch: opened by test main via fcntl dup? -> fallback db) */
return 0;
}
static int c_owner_misc(void)
{
/* nothing here may be flagged */
copy_like(P("regtest/wallet.dat")); /* common-caller */
int fd = __os_openhandle(P("regtest/blocks/blk00003.dat"));
FILE *f = fdopen(fd, "rb+"); /* ownership moves to FILE */
FlushBlockFile(f);
int s = ConnectSocketDirectly(); CloseSocket(&s);
fd = (int)syscall(SYS_openat, AT_FDCWD, P("regtest/wallet.dat"), O_RDWR); /* untracked open */
__os_closehandle(fd); /* fallback:bdb */
return 0;
}
static int c_fallback_hit(void)
{
int fd = (int)syscall(SYS_openat, AT_FDCWD, P("regtest/wallet.dat"), O_RDWR); /* untracked open */
foreign_stale_close(fd); /* HIT via fallback rule */
return 0;
}
static int c_epoll(void)
{
int ep = epoll_create1(0);
int fd = plain_open(P("plain2.txt"));
struct epoll_event ev = { .events = EPOLLIN };
int r = epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev); /* EPERM: regular file */
close(fd);
r = epoll_ctl(ep, EPOLL_CTL_ADD, fd, &ev); /* EBADF */
close(ep);
return r == -1 ? 0 : 1;
}
static int c_selfclose(void)
{
/* close every fd from 3 to 1100 by number, like a buggy daemonizer */
for (int i = 3; i < 1100; i++) close(i);
return 0;
}
/* concurrency + micro-benchmark: open/close churn in many threads */
static int nthr = 8, iters = 20000;
static void *worker(void *arg)
{
long id = (long)arg;
char p[700]; snprintf(p, sizeof p, "%s/thr%ld.txt", D, id);
for (int i = 0; i < iters; i++) {
int s = ConnectSocketDirectly(); CloseSocket(&s);
int fd = plain_open(p); plain_close(fd);
FILE *f = fopen(p, "r"); if (f) fclose(f);
if ((i & 255) == 0) { int w = __os_openhandle(P("regtest/wallet.dat")); __os_closehandle(w); }
}
return NULL;
}
static int c_threads(void)
{
const char *e = getenv("T_ITERS"); if (e) iters = atoi(e);
pthread_t t[64];
struct timespec a, b; clock_gettime(CLOCK_MONOTONIC, &a);
for (long i = 0; i < nthr; i++) pthread_create(&t[i], NULL, worker, (void *)i);
for (int i = 0; i < nthr; i++) pthread_join(t[i], NULL);
clock_gettime(CLOCK_MONOTONIC, &b);
double ns = (b.tv_sec - a.tv_sec) * 1e9 + (b.tv_nsec - a.tv_nsec);
long ops = (long)nthr * iters * 3;
printf("threads=%d iters=%d ops=%ld (socket/open/fopen + close each) wall=%.3fs %.0f ns/op-pair/thread\n",
nthr, iters, ops, ns / 1e9, ns * nthr / ops);
return 0;
}
int main(int argc, char **argv)
{
if (argc < 3) { fprintf(stderr, "usage: %s <dir> <case>\n", argv[0]); return 2; }
snprintf(D, sizeof D, "%s", argv[1]);
const char *c = argv[2];
if (!strcmp(c, "double")) return c_double();
if (!strcmp(c, "dbowner")) return c_dbowner();
if (!strcmp(c, "dbforeign")) return c_dbforeign();
if (!strcmp(c, "deny")) return c_deny();
if (!strcmp(c, "blkowner")) return c_blkowner();
if (!strcmp(c, "blkraw")) return c_blkraw();
if (!strcmp(c, "stalefile")) return c_stalefile();
if (!strcmp(c, "reuse")) return c_reuse();
if (!strcmp(c, "dup2")) return c_dup2();
if (!strcmp(c, "range")) return c_range();
if (!strcmp(c, "ownermisc")) return c_owner_misc();
if (!strcmp(c, "fallback")) return c_fallback_hit();
if (!strcmp(c, "epoll")) return c_epoll();
if (!strcmp(c, "selfclose")) return c_selfclose();
if (!strcmp(c, "threads")) return c_threads();
fprintf(stderr, "unknown case %s\n", c);
return 2;
}
qa/fd-guard/shim/run_tests.sh
Executable
+97
View File
@@ -0,0 +1,97 @@
#!/bin/bash
# Self-test for libfdguard.so. Every case lists the report types it must
# produce (exactly, in order) -- an empty list means "no report at all".
# ./run_tests.sh (builds first)
set -u
cd "$(dirname "$0")"
./build.sh >/dev/null || { echo "build failed"; exit 1; }
SO=$PWD/libfdguard.so
T=$PWD/t
rm -rf "$T"; mkdir -p "$T/regtest/blocks" "$T/regtest/database"
pass=0; fail=0
run() { # run <case> [ENV=VAL ...] -> sets RC, LOG
local c=$1; shift
LOG=$T/$c${SUFFIX:-}.log; rm -f "$LOG"
env FDGUARD_EXE=fdguard_test FDGUARD_LOG="$LOG" "$@" LD_PRELOAD="$SO" ./fdguard_test "$T" "$c" >"$T/$c${SUFFIX:-}.out" 2>&1
RC=$?
}
types() { grep '^=== FDGUARD' "$LOG" 2>/dev/null | awk '{print $4}' | tr '\n' ' ' | sed 's/ $//'; }
reasons() { grep -o 'reason=[^ ]*' "$LOG" 2>/dev/null | sed 's/reason=//' | tr '\n' ' ' | sed 's/ $//'; }
check() { # check <name> <cond-description> <actual> <expected>
if [ "$3" == "$4" ]; then pass=$((pass+1)); printf 'PASS %-12s %s\n' "$1" "$2";
else fail=$((fail+1)); printf 'FAIL %-12s %s\n got: [%s]\n want: [%s]\n' "$1" "$2" "$3" "$4"; fi
}
# case | expected report types | expected reasons
while IFS='|' read -r c want wantr; do
[ -z "$c" ] && continue
run "$c"
check "$c" "reports" "$(types)" "$want"
check "$c" "reasons" "$(reasons)" "$wantr"
check "$c" "exit 0" "$RC" "0"
done <<'EOF'
double|EBADF-CLOSE|
dbowner||
dbforeign|NON-OWNER-CLOSE EBADF-CLOSE|DOMAIN-MISMATCH
blkowner||
blkraw|NON-OWNER-CLOSE EBADF-CLOSE|stdio:RAW-CLOSE-UNDER-LIVE-FILE
stalefile|NON-OWNER-CLOSE NON-OWNER-CLOSE EBADF-CLOSE|stdio:RAW-CLOSE-UNDER-LIVE-FILE stdio:STALE-FILE
reuse|NON-OWNER-CLOSE EBADF-CLOSE|DOMAIN-MISMATCH
dup2|NON-OWNER-CLOSE|DOMAIN-MISMATCH
range|NON-OWNER-CLOSE NON-OWNER-CLOSE NON-OWNER-CLOSE|DOMAIN-MISMATCH DOMAIN-MISMATCH fallback:DB-FILE-CLOSED-OUTSIDE-BDB
ownermisc||
fallback|NON-OWNER-CLOSE|fallback:DB-FILE-CLOSED-OUTSIDE-BDB
epoll|EPOLL-CTL-EPERM EPOLL-CTL-EBADF|epoll_ctl(ADD)-failed epoll_ctl(ADD)-failed
EOF
# the previous close of the reused number must be the first CloseSocket (proves the double close)
run reuse
check reuse "history shows 1st CloseSocket on a socket" \
"$(grep -A2 'previous successful close' "$LOG" | head -3 | grep -c -e 'class=sock dom=p2p' -e 'CloseSocket+')" "2"
# the stale FILE report must name wallet.dat as the victim and OpenDiskFile as the FILE's opener
run stalefile
check stalefile "victim=wallet.dat, FILE opened in OpenDiskFile" \
"$(sed -n '/#2 NON-OWNER/,/=== END #2/p' "$LOG" | grep -m1 -c 'target=.*wallet.dat class=db') $(sed -n '/#2 NON-OWNER/,/=== END #2/p' "$LOG" | grep -A3 'which had been opened by: fopen' | grep -c 'OpenDiskFile+')" "1 1"
# the log fd survives a close-everything loop and the attempt is reported
run selfclose
check selfclose "log-fd close refused + reported" "$(grep -c 'CLOSE-OF-FDGUARD-LOG-FD' "$LOG")" "1"
check selfclose "log still writable afterwards (final stats)" "$(grep -c '^S .* final' "$LOG")" "1"
# abort mode: first non-owner close -> SIGABRT (rc 134), report already written
SUFFIX=_abort run dbforeign FDGUARD_ABORT=1
check abort "rc=134 (SIGABRT)" "$RC" "134"
check abort "report written before abort" "$(types)" "NON-OWNER-CLOSE"
# abort=2 also aborts on EBADF
SUFFIX=_abort2 run double FDGUARD_ABORT=2
check abort2 "rc=134 on EBADF" "$RC" "134"
# deny mode: wallet fd survives the foreign close; no EBADF afterwards
SUFFIX=_deny run deny FDGUARD_DENY=1
check deny "exit 0 (fd still valid + writable)" "$RC" "0"
check deny "reported as DENIED, no EBADF" "$(types) $(grep -c 'ACTION=DENIED' "$LOG")" "NON-OWNER-CLOSE 1"
# extra classes: guard=all with the owner cases must still be silent
SUFFIX=_all run ownermisc FDGUARD_GUARD=all
check guard-all "ownermisc silent with FDGUARD_GUARD=all" "$(types)" ""
# trace-all writes one T line per open/close
SUFFIX=_trace run dbowner FDGUARD_TRACE_ALL=1
check trace "T lines for open+close" "$(grep -c '^T ' "$LOG")" "2"
# passive in non-matching processes
LOG=$T/passive.log; env FDGUARD_EXE=dragonxd FDGUARD_LOG="$LOG" LD_PRELOAD="$SO" ./fdguard_test "$T" dbforeign >/dev/null 2>&1
check passive "no log when exe does not match" "$( [ -e "$LOG" ] && echo exists || echo none)" "none"
# concurrency: 8 threads x 5000 x (socket, open, fopen) churn, guard=all -> no reports
SUFFIX=_thr run threads T_ITERS=5000 FDGUARD_GUARD=all
check threads "no reports under 8-thread churn (guard=all)" "$(types)" ""
check threads "stats: HITS=0 EBADF=0" "$(grep '^S .*final' "$LOG" | grep -o 'HITS=[0-9]* EBADF=[0-9]*')" "HITS=0 EBADF=0"
# symbolizer resolves file:line for the test binary
check symbolize "addr2line resolved CloseSocket to fdguard_test.c" \
"$(./symbolize.py "$T/reuse.log" | grep -m1 -o 'in CloseSocket at .*fdguard_test.c:[0-9]*' | sed 's/.*fdguard_test.c:[0-9]*/ok/')" "ok"
echo "== $pass passed, $fail failed"
[ $fail -eq 0 ]
qa/fd-guard/shim/symbolize.py
Executable
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env python3
"""
symbolize.py -- turn fdguard log frames into function + file:line.
Every stack frame in an fdguard report looks like
#3 0x5b96b433c991 /path/to/module+0x1991 c_dbforeign+0x2d [bdb]
where "+0x1991" is the ELF virtual address inside that module (runtime address
minus the module's load bias, so PIE executables and shared libraries need no
further adjustment; for a non-PIE executable it equals the absolute address).
Frames are return addresses, so we ask addr2line about vaddr-1 (the call
instruction), with -f -C -i (function, demangle, inline chain).
backtrace_symbols_fd lines "module(+0x1991)[0x...]" are symbolized too.
usage:
symbolize.py [options] LOG [LOG...] annotated log on stdout
symbolize.py --summary LOG one line per distinct report
options:
--map OLD=NEW read module OLD from path NEW instead (repeatable),
e.g. --map /path/to/deployed/dragonxd=./dragonxd.unstripped
--addr2line PATH addr2line binary (default: addr2line)
--only-reports drop T/S lines (trace + stats) from the output
"""
import os
import re
import subprocess
import sys
from collections import defaultdict, OrderedDict
FRAME_RE = re.compile(r'^(\s*#\d+ 0x[0-9a-f]+ )(\S+)\+0x([0-9a-f]+)(.*)$')
BSFD_RE = re.compile(r'^(\S+)\(\+0x([0-9a-f]+)\)\[0x[0-9a-f]+\]\s*$')
HDR_RE = re.compile(r'^=== FDGUARD #(\d+) (\S+)')
def parse_args(argv):
opts = {'map': {}, 'addr2line': 'addr2line', 'summary': False, 'only': False, 'logs': []}
it = iter(argv)
for a in it:
if a == '--map':
old, new = next(it).split('=', 1)
opts['map'][old] = new
elif a.startswith('--map='):
old, new = a[6:].split('=', 1)
opts['map'][old] = new
elif a == '--addr2line':
opts['addr2line'] = next(it)
elif a == '--summary':
opts['summary'] = True
elif a == '--only-reports':
opts['only'] = True
elif a in ('-h', '--help'):
print(__doc__)
sys.exit(0)
else:
opts['logs'].append(a)
if not opts['logs']:
opts['logs'] = ['-']
return opts
def read_lines(paths):
out = []
for p in paths:
f = sys.stdin if p == '-' else open(p, errors='replace')
out.extend(f.read().splitlines())
if f is not sys.stdin:
f.close()
return out
def resolve_module(mod, opts, known):
if mod in opts['map']:
return opts['map'][mod]
if os.path.exists(mod):
return mod
base = os.path.basename(mod)
for k in known:
if os.path.basename(k) == base and os.path.exists(opts['map'].get(k, k)):
return opts['map'].get(k, k)
return None
def run_addr2line(binary, addrs, a2l):
"""addrs: list of int vaddrs (already -1'd). returns {addr: [(func, loc), ...]}"""
res = {}
if not addrs:
return res
uniq = sorted(set(addrs))
try:
p = subprocess.run([a2l, '-a', '-f', '-C', '-i', '-e', binary] + ['0x%x' % a for a in uniq],
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, check=False)
except FileNotFoundError:
return res
cur = None
lines = p.stdout.splitlines()
i = 0
while i < len(lines):
ln = lines[i]
if re.match(r'^0x[0-9a-f]+$', ln):
cur = int(ln, 16)
res[cur] = []
i += 1
continue
func = ln
loc = lines[i + 1] if i + 1 < len(lines) else '??:0'
if cur is not None:
res[cur].append((func, loc))
i += 2
return res
def main():
opts = parse_args(sys.argv[1:])
lines = read_lines(opts['logs'])
if opts['summary']:
summary(lines)
return
# pass 1: collect modules + addresses
want = defaultdict(set)
known = OrderedDict()
for ln in lines:
m = FRAME_RE.match(ln)
if m:
known[m.group(2)] = 1
want[m.group(2)].add(max(int(m.group(3), 16) - 1, 0))
for ln in lines:
m = BSFD_RE.match(ln)
if m:
want[m.group(1)].add(max(int(m.group(2), 16) - 1, 0))
table = {}
for mod, addrs in want.items():
path = resolve_module(mod, opts, known)
if not path:
continue
table[mod] = run_addr2line(path, list(addrs), opts['addr2line'])
def fmt(mod, vaddr):
r = table.get(mod, {}).get(vaddr)
if not r:
return []
out = []
for j, (func, loc) in enumerate(r):
if func == '??' and loc.startswith('??'):
continue
# addr2line -i lists the innermost (inlined) frame first
tag = 'inlined:' if j < len(r) - 1 else 'in'
out.append('%s %s at %s' % (tag, func, loc))
return out
w = sys.stdout.write
for ln in lines:
if opts['only'] and (ln.startswith('T ') or ln.startswith('S ')):
continue
w(ln + '\n')
m = FRAME_RE.match(ln)
if m:
v = max(int(m.group(3), 16) - 1, 0)
for s in fmt(m.group(2), v):
w(' -> ' + s + '\n')
continue
m = BSFD_RE.match(ln)
if m:
v = max(int(m.group(2), 16) - 1, 0)
for s in fmt(m.group(1), v):
w(' -> ' + s + '\n')
def summary(lines):
"""one line per (type, reason, class, first exe closer frame)"""
groups = OrderedDict()
cur = None
in_closer = False
for ln in lines:
h = HDR_RE.match(ln)
if h:
cur = {'type': h.group(2), 'reason': '', 'cls': '', 'top': '', 'call': ''}
in_closer = False
continue
if cur is None:
continue
s = ln.strip()
if s.startswith('call:'):
cur['call'] = s.split('(')[0][6:]
elif s.startswith('target=') and ' class=' in s and not cur['cls']:
cur['cls'] = s.rsplit('class=', 1)[-1]
elif s.startswith('verdict='):
cur['reason'] = s.split('reason=', 1)[-1].split(' ')[0]
elif s == 'closer stack:':
in_closer = True
elif in_closer and FRAME_RE.match(ln):
parts = s.split(' ')
if len(parts) >= 4 and not cur['top'] and '/libc.so' not in parts[2] and '/libstdc++' not in parts[2]:
cur['top'] = parts[3]
elif in_closer and not s.startswith('#'):
in_closer = False
if s.startswith('=== END'):
k = (cur['type'], cur['call'], cur['reason'], cur['cls'], cur['top'])
groups[k] = groups.get(k, 0) + 1
cur = None
print('%6s %-22s %-12s %-6s %-40s %s' % ('count', 'type', 'call', 'class', 'reason', 'first closer frame'))
for (t, c, r, cl, top), n in groups.items():
print('%6d %-22s %-12s %-6s %-40s %s' % (n, t[:22], c, cl, r[:40], top))
stats = [ln for ln in lines if ln.startswith('S ')]
if stats:
print('\nlast stats line:\n ' + stats[-1])
if __name__ == '__main__':
main()
src/Makefile.gtest.include
+7 -1
View File
@@ -14,7 +14,13 @@ hush_gtest_SOURCES = \
gtest/test_randomx_preverify.cpp \
gtest/test_hdtransparent.cpp \
gtest/test_mnemonic_compat.cpp \
gtest/test_stratum_jobid.cpp
gtest/test_stratum_jobid.cpp \
gtest/test_wallet_notedata.cpp \
gtest/test_net_closesocket.cpp \
gtest/test_fd_defense.cpp \
gtest/test_wallet_dbenv.cpp \
gtest/test_wallet_notecache.cpp \
gtest/test_wallet_history.cpp
hush_gtest_CPPFLAGS = $(AM_CPPFLAGS) -DMULTICORE -fopenmp -DBINARY_OUTPUT -DCURVE_ALT_BN128 -DSTATIC $(BITCOIN_INCLUDES)
hush_gtest_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
src/gtest/test_fd_defense.cpp
+83
View File
@@ -0,0 +1,83 @@
// Copyright (c) 2016-2024 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
#include <gtest/gtest.h>
#include <atomic>
#include <errno.h>
#include <stdexcept>
#include <stdio.h>
#include <unistd.h>
#include <boost/filesystem.hpp>
#include <boost/thread.hpp>
#include "chainparams.h"
#include "init.h"
#include "main.h"
#include "noui.h"
#include "util.h"
#include "validationinterface.h"
extern std::atomic<bool> fRequestShutdown;
// A block file opened for reading must not be writable through its fd: a stray write through a reused fd number
// (the wolfSSL CloseSocket macro let the wallet's BDB fd be closed behind its back) used to land in it.
TEST(FdDefense, ReadOnlyBlockFileIsNotWritable) {
SelectParams(CBaseChainParams::MAIN);
const boost::filesystem::path dir = boost::filesystem::temp_directory_path() / boost::filesystem::unique_path("fddefense-%%%%%%%%");
const bool hadDatadir = mapArgs.count("-datadir") > 0;
const std::string oldDatadir = hadDatadir ? mapArgs["-datadir"] : "";
boost::filesystem::create_directories(dir);
mapArgs["-datadir"] = dir.string();
ClearDatadirCache();
const CDiskBlockPos pos(0, 0);
const boost::filesystem::path blk = GetBlockPosFilename(pos, "blk");
boost::filesystem::create_directories(blk.parent_path());
{
FILE* f = fopen(blk.string().c_str(), "wb");
ASSERT_TRUE(f != nullptr);
fputs("block bytes", f);
fclose(f);
}
FILE* ro = OpenBlockFile(pos, true);
ASSERT_TRUE(ro != nullptr);
char c = 0;
EXPECT_EQ(1u, fread(&c, 1, 1, ro)) << "still readable";
errno = 0;
EXPECT_EQ(-1, pwrite(fileno(ro), "X", 1, 0)) << "a write through a read-only block file's fd must fail";
EXPECT_EQ(EBADF, errno);
fclose(ro);
FILE* rw = OpenBlockFile(pos, false);
ASSERT_TRUE(rw != nullptr);
EXPECT_EQ(1, pwrite(fileno(rw), "b", 1, 0)) << "the writable open still writes";
fclose(rw);
if (hadDatadir) mapArgs["-datadir"] = oldDatadir; else mapArgs.erase("-datadir");
ClearDatadirCache();
boost::filesystem::remove_all(dir);
}
// A wallet notification that throws (a wallet DB failure) requests a clean shutdown instead of escaping the
// notifier thread into std::terminate; a successful one doesn't; thread interruption still propagates.
TEST(FdDefense, WalletNotifyFailureShutsDownCleanly) {
noui_connect(); // the daemon connects these UI signals at startup; the error path shows a message box
fRequestShutdown = false;
EXPECT_TRUE(NotifyWalletsOrShutdown([] {}));
EXPECT_FALSE(ShutdownRequested());
EXPECT_FALSE(NotifyWalletsOrShutdown([] { throw std::runtime_error("CWalletDB::ListAccountCreditDebit(): error scanning DB"); }));
EXPECT_TRUE(ShutdownRequested());
fRequestShutdown = false;
EXPECT_FALSE(NotifyWalletsOrShutdown([] { throw 42; }));
EXPECT_TRUE(ShutdownRequested());
fRequestShutdown = false;
EXPECT_THROW(NotifyWalletsOrShutdown([] { throw boost::thread_interrupted(); }), boost::thread_interrupted);
EXPECT_FALSE(ShutdownRequested());
}
src/gtest/test_net_closesocket.cpp
+50
View File
@@ -0,0 +1,50 @@
// Copyright (c) 2016-2024 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
#include <gtest/gtest.h>
#include <fcntl.h>
#include <sys/socket.h>
#include <unistd.h>
#include "net.h"
#include "util/sock.h"
// Regression test for the wolfSSL CloseSocket macro (see net.h): closing a CNode's socket must invalidate
// hSocket, so neither a repeated CloseSocketDisconnect nor ~CNode can close that fd number again once another
// owner (the wallet's BDB files, a block file, an RPC socket) has been handed it.
TEST(NetCloseSocket, ClosedNodeNeverClosesAReusedFdNumber) {
int pair[2];
ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_STREAM, 0, pair));
const int number = pair[0];
CNode* node = new CNode(number, CAddress(), "", true); // inbound: sends no version message
node->CloseSocketDisconnect();
EXPECT_EQ(INVALID_SOCKET, node->hSocket) << "CloseSocketDisconnect left the closed fd number in hSocket";
EXPECT_EQ(-1, fcntl(number, F_GETFD)) << "the node's socket wasn't closed";
// Another owner gets the freed number, as the kernel's lowest-free-fd rule does in a busy node.
const int other = open("/dev/null", O_RDONLY);
ASSERT_GE(other, 0);
ASSERT_EQ(number, dup2(other, number));
node->CloseSocketDisconnect(); // e.g. a send error after the disconnect was already handled
delete node; // ~CNode
EXPECT_NE(-1, fcntl(number, F_GETFD)) << "a closed CNode closed the fd number's new owner";
close(number);
close(other);
close(pair[1]);
}
// util/sock.cpp's CloseSocket is the one that must be called: it closes once and invalidates.
TEST(NetCloseSocket, CloseSocketInvalidates) {
int pair[2];
ASSERT_EQ(0, socketpair(AF_UNIX, SOCK_STREAM, 0, pair));
SOCKET s = pair[0];
EXPECT_TRUE(CloseSocket(s));
EXPECT_EQ(INVALID_SOCKET, s);
EXPECT_FALSE(CloseSocket(s)) << "a second close of an invalidated handle must be a no-op";
close(pair[1]);
}
src/gtest/test_wallet_dbenv.cpp
+58
View File
@@ -0,0 +1,58 @@
// Copyright (c) 2016-2024 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
#include <gtest/gtest.h>
#include <boost/filesystem.hpp>
#include "wallet/db.h"
namespace {
// A fresh environment in a temp dir with one committed record in "wallet.dat", so database/ holds a log file.
class WalletDbEnv : public ::testing::Test {
protected:
boost::filesystem::path dir;
CDBEnv env;
void SetUp() override {
dir = boost::filesystem::temp_directory_path() / boost::filesystem::unique_path("dbenv-%%%%%%%%");
boost::filesystem::create_directories(dir);
ASSERT_TRUE(env.Open(dir));
Db db(env.dbenv, DB_CXX_NO_EXCEPTIONS);
ASSERT_EQ(0, db.open(NULL, "wallet.dat", "main", DB_BTREE, DB_CREATE | DB_AUTO_COMMIT, 0));
char k[] = "key", v[] = "value";
Dbt key(k, sizeof k), value(v, sizeof v);
ASSERT_EQ(0, db.put(NULL, &key, &value, 0));
ASSERT_EQ(0, db.close(0));
ASSERT_TRUE(boost::filesystem::exists(Log()));
}
void TearDown() override {
env.Close();
boost::system::error_code ec;
boost::filesystem::remove_all(dir, ec);
}
boost::filesystem::path Log() const { return dir / "database" / "log.0000000001"; }
};
} // namespace
TEST_F(WalletDbEnv, CleanShutdownRemovesTheLogs) {
env.Flush(true);
EXPECT_FALSE(boost::filesystem::exists(dir / "database"));
}
// BDB0060 PANIC (a page write failed, e.g. through an fd closed behind BDB's back): the logs may hold committed
// changes wallet.dat never got, and the next start's DB_RECOVER needs them.
TEST_F(WalletDbEnv, PanickedEnvironmentKeepsTheLogs) {
ASSERT_EQ(0, env.dbenv->set_flags(DB_PANIC_ENVIRONMENT, 1));
env.Flush(true);
EXPECT_TRUE(boost::filesystem::exists(Log()));
}
TEST_F(WalletDbEnv, FailedCheckpointKeepsTheLogs) {
env.CheckpointLSN("no-such-file.dat"); // lsn_reset fails
env.Flush(true);
EXPECT_TRUE(boost::filesystem::exists(Log()));
}
src/gtest/test_wallet_history.cpp
+816
View File
@@ -0,0 +1,816 @@
// Copyright (c) 2016-2024 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
#include <gtest/gtest.h>
#include <deque>
#include <utf8.h>
#include "chain.h"
#include "chainparams.h"
#include "coins.h"
#include "init.h"
#include "key_io.h"
#include "main.h"
#include "primitives/block.h"
#include "primitives/transaction.h"
#include "random.h"
#include "rpc/server.h"
#include "script/standard.h"
#include "utilmoneystr.h"
#include "wallet/wallet.h"
#include "wallet/rpchushwallet.h"
#include "zcash/Address.hpp"
#include "zcash/Note.hpp"
#include "zcash/NoteEncryption.hpp"
#include "zcash/zip32.h"
using namespace std;
using namespace libzcash;
// The history-RPC helpers (src/wallet/rpchushwallet.cpp) as of 69bb035e7, verbatim apart from the Legacy_
// prefix and static linkage: the reference the optimized helpers must reproduce byte for byte.
static void Legacy_zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) {
LOCK2(cs_main, pwalletMain->cs_wallet);
//Used to identify incomplete key sets
int vinCount = 0;
int vinSpendCount = 0;
int shieldedSpendCount = 0;
int shieldedSpendsSpentCount = 0;
//Check address
bool isTAddress = false;
bool isZsAddress = false;
CTxDestination tAddress = DecodeDestination(strAddress);
auto zAddress = DecodePaymentAddress(strAddress);
SaplingPaymentAddress zsAddress;
if (filterByAddress) {
if (IsValidDestination(tAddress))
isTAddress = true;
if (IsValidPaymentAddress(zAddress)) {
if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
isZsAddress = true;
}
}
}
// Transparent Inputs belonging to the wallet
UniValue tSpends(UniValue::VARR);
if (isTAddress || !filterByAddress) {
for (int i = 0; i < wtx.vin.size(); i++) {
vinCount++;
const CTxIn& txin = wtx.vin[i];
UniValue obj(UniValue::VOBJ);
CTxDestination address;
const CWalletTx* parent = pwalletMain->GetWalletTx(txin.prevout.hash);
if (parent != NULL) {
const CTxOut& parentOut = parent->vout[txin.prevout.n];
ExtractDestination(parentOut.scriptPubKey, address);
if(IsMine(*pwalletMain, address)){
vinSpendCount++;
totalSpends += CAmount(-parentOut.nValue);
obj.push_back(Pair("address",EncodeDestination(address)));
obj.push_back(Pair("scriptPubKey",HexStr(parentOut.scriptPubKey.begin(), parentOut.scriptPubKey.end())));
obj.push_back(Pair("amount",ValueFromAmount(-parentOut.nValue)));
obj.push_back(Pair("spendTxid",parent->GetHash().ToString()));
obj.push_back(Pair("spendVout",(int)txin.prevout.n));
CTxDestination filterAddress = DecodeDestination(strAddress);
if (address == tAddress || !filterByAddress) {
filteredSpends += CAmount(-parentOut.nValue);
tSpends.push_back(obj);
}
}
}
}
}
spends.push_back(Pair("transparentSpends",tSpends));
// Sapling Inputs belonging to the wallet
UniValue zsSpends(UniValue::VARR);
if (isZsAddress || !filterByAddress) {
for (int i = 0; i < wtx.vShieldedSpend.size(); i++) {
shieldedSpendCount++;
const SpendDescription& spendDesc = wtx.vShieldedSpend[i];
UniValue obj(UniValue::VOBJ);
SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier];
if (pwalletMain->IsSaplingNullifierFromMe(spendDesc.nullifier)) {
const CWalletTx* parent = pwalletMain->GetWalletTx(pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash);
const OutputDescription& output = parent->vShieldedOutput[op.n];
auto nd = pwalletMain->mapWallet[pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash].mapSaplingNoteData[op];
auto pt = libzcash::SaplingNotePlaintext::decrypt(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
if (pt) {
auto note = pt.get();
auto pa = nd.ivk.address(note.d);
auto address = pa.get();
shieldedSpendsSpentCount++;
totalSpends += CAmount(-note.value());
obj.push_back(Pair("address",EncodePaymentAddress(address)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(-note.value()))));
obj.push_back(Pair("spendTxid",parent->GetHash().ToString()));
obj.push_back(Pair("spendshieldedOutputIndex",(int)op.n));
if (address == zsAddress || !filterByAddress) {
filteredSpends += CAmount(-note.value());
zsSpends.push_back(obj);
}
}
}
}
}
spends.push_back(Pair("saplingSpends",zsSpends));
spends.push_back(Pair("totalSpends",ValueFromAmount(filteredSpends)));
if (!filterByAddress) {
if (filteredSpends != 0 && (vinCount != vinSpendCount || shieldedSpendCount != shieldedSpendsSpentCount)) {
spends.push_back(Pair("missingSpendingKeys", true));
spends.push_back(Pair("vinCount", vinCount));
spends.push_back(Pair("vinSpendCount", vinSpendCount));
spends.push_back(Pair("shieldedSpendCount", shieldedSpendCount));
spends.push_back(Pair("shieldedSpendsSpentCount", shieldedSpendsSpentCount));
} else {
spends.push_back(Pair("missingSpendingKeys", false));
}
}
}
static void Legacy_zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress) {
if(fZdebug)
fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
LOCK2(cs_main, pwalletMain->cs_wallet);
//Check address
bool isTAddress = false;
bool isZsAddress = false;
CTxDestination tAddress = DecodeDestination(strAddress);
auto zAddress = DecodePaymentAddress(strAddress);
SaplingPaymentAddress zsAddress;
if (filterByAddress) {
if (IsValidDestination(tAddress))
isTAddress = true;
if (IsValidPaymentAddress(zAddress)) {
if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
isZsAddress = true;
}
}
}
//Transparent Received txos belonging to the wallet
UniValue tReceived(UniValue::VARR);
if (isTAddress || !filterByAddress) {
for (int i = 0; i < wtx.vout.size(); i++) {
const CTxOut& txout = wtx.vout[i];
UniValue obj(UniValue::VOBJ);
CTxDestination address;
ExtractDestination(txout.scriptPubKey, address);
if(IsMine(*pwalletMain, address)){
obj.push_back(Pair("address",EncodeDestination(address)));
obj.push_back(Pair("scriptPubKey",HexStr(txout.scriptPubKey.begin(), txout.scriptPubKey.end())));
obj.push_back(Pair("amount",ValueFromAmount(txout.nValue)));
obj.push_back(Pair("vout", i));
if (address == tAddress || !filterByAddress) {
totalReceived += CAmount(txout.nValue);
tReceived.push_back(obj);
}
}
}
}
received.push_back(Pair("transparentReceived",tReceived));
//Sapling Sends belonging to the wallet
UniValue zsReceived(UniValue::VARR);
if (isZsAddress || !filterByAddress) {
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
UniValue obj(UniValue::VOBJ);
bool changeTx = false;
//Decrypt sapling incoming commitments using IVK
std::set<libzcash::SaplingPaymentAddress> addresses;
pwalletMain->GetSaplingPaymentAddresses(addresses);
for (auto addr : addresses) {
libzcash::SaplingExtendedSpendingKey extsk;
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext, extsk.expsk.full_viewing_key().in_viewing_key(), outputDesc.ephemeralKey, outputDesc.cm);
if (pt) {
auto note = pt.get();
auto pt_unwrapped = pt.get();
auto memo = pt_unwrapped.memo();
obj.push_back(Pair("address",EncodePaymentAddress(addr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value()))));
obj.push_back(Pair("memo", HexStr(memo)));
if (memo[0] <= 0xf4) {
auto end = std::find_if(memo.rbegin(), memo.rend(), [](unsigned char v) { return v != 0; });
std::string memoStr(memo.begin(), end.base());
if (utf8::is_valid(memoStr)) {
obj.push_back(Pair("memoStr", memoStr));
}
}
obj.push_back(Pair("shieldedOutputIndex",i));
//Check Change Status
if (wtx.vShieldedSpend.size()!=0) {
std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
nullifierSet = pwalletMain->GetNullifiersForAddresses({addr});
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
if (nullifierSet.count(std::make_pair(addr, spendDesc.nullifier))) {
changeTx = true;
}
}
}
obj.push_back(Pair("change",changeTx));
if (addr == zsAddress || !filterByAddress) {
totalReceived += CAmount(note.value());
zsReceived.push_back(obj);
}
}
}
}
}
}
received.push_back(Pair("saplingReceived",zsReceived));
received.push_back(Pair("totalReceived",ValueFromAmount(totalReceived)));
}
static void Legacy_zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress) {
LOCK2(cs_main, pwalletMain->cs_wallet);
//Used to identify incomplete key sets
int shieldedOutputCount = 0;
int shieldedOutputDecryptedCount = 0;
//Check address
bool isTAddress = false;
bool isZsAddress = false;
CTxDestination tAddress = DecodeDestination(strAddress);
auto zAddress = DecodePaymentAddress(strAddress);
SaplingPaymentAddress zsAddress;
if (filterByAddress) {
if (IsValidDestination(tAddress))
isTAddress = true;
if (IsValidPaymentAddress(zAddress)) {
if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
isZsAddress = true;
}
}
}
//All Transparent Sends in the transaction
UniValue tSends(UniValue::VARR);
if (isTAddress || !filterByAddress) {
for (int i = 0; i < wtx.vout.size(); i++) {
const CTxOut& txout = wtx.vout[i];
UniValue obj(UniValue::VOBJ);
CTxDestination address;
ExtractDestination(txout.scriptPubKey, address);
obj.push_back(Pair("address",EncodeDestination(address)));
obj.push_back(Pair("scriptPubKey",HexStr(txout.scriptPubKey.begin(), txout.scriptPubKey.end())));
obj.push_back(Pair("amount",ValueFromAmount(-txout.nValue)));
obj.push_back(Pair("vout", i));
if (address == tAddress || !filterByAddress) {
totalSends += CAmount(-txout.nValue);
tSends.push_back(obj);
}
}
}
sends.push_back(Pair("transparentSends",tSends));
//All Shielded Sends in the transaction
UniValue zsSends(UniValue::VARR);
if (isZsAddress || !filterByAddress) {
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
shieldedOutputCount++;
UniValue obj(UniValue::VOBJ);
bool changeTx = false;
//Decrypt sapling outgoing t to z transaction using HDseed
if (wtx.vShieldedSpend.size()==0) {
HDSeed seed;
if (pwalletMain->GetHDSeedForDerivation(seed)) {
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
outputDesc.outCiphertext,ovkForShieldingFromTaddr(seed),outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
if (opt) {
auto opt_unwrapped = opt.get();
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
if (pt) {
shieldedOutputDecryptedCount++;
auto pt_unwrapped = pt.get();
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value()))));
obj.push_back(Pair("shieldedOutputIndex",i));
obj.push_back(Pair("change",false));
if (sentAddr == zsAddress || !filterByAddress) {
totalSends += CAmount(-pt_unwrapped.value());
zsSends.push_back(obj);
}
}
}
}
//attempt Decryption of Outgoing Sapling using wallet extended spending keys
} else {
std::set<libzcash::SaplingPaymentAddress> addresses;
pwalletMain->GetSaplingPaymentAddresses(addresses);
for (auto addr : addresses) {
libzcash::SaplingExtendedSpendingKey extsk;
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
outputDesc.outCiphertext,extsk.expsk.full_viewing_key().ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
if (opt) {
auto opt_unwrapped = opt.get();
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
if (pt) {
auto pt_unwrapped = pt.get();
auto memo = pt_unwrapped.memo();
shieldedOutputDecryptedCount++;
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value()))));
obj.push_back(Pair("memo", HexStr(memo)));
if (memo[0] <= 0xf4) {
auto end = std::find_if(memo.rbegin(), memo.rend(), [](unsigned char v) { return v != 0; });
std::string memoStr(memo.begin(), end.base());
if (utf8::is_valid(memoStr)) {
obj.push_back(Pair("memoStr", memoStr));
}
}
obj.push_back(Pair("shieldedOutputIndex",i));
//Check Change Status
if (wtx.vShieldedSpend.size()!=0) {
std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
nullifierSet = pwalletMain->GetNullifiersForAddresses({sentAddr});
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
if (nullifierSet.count(std::make_pair(sentAddr, spendDesc.nullifier))) {
changeTx = true;
}
}
}
obj.push_back(Pair("change",changeTx));
if (sentAddr == zsAddress || !filterByAddress) {
totalSends += CAmount(-pt_unwrapped.value());
zsSends.push_back(obj);
}
}
}
}
}
}
}
}
sends.push_back(Pair("saplingSends",zsSends));
if (shieldedOutputCount != shieldedOutputDecryptedCount) {
sends.push_back(Pair("missingSaplingOVK", true));
} else {
sends.push_back(Pair("missingSaplingOVK", false));
}
sends.push_back(Pair("totalSends",ValueFromAmount(totalSends)));
}
static void Legacy_zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType) {
LOCK2(cs_main, pwalletMain->cs_wallet);
//Track total wallet spend and received
CAmount totalSpends = 0;
CAmount filteredSpends = 0;
CAmount totalReceived = 0;
CAmount totalSends = 0;
//Various Univalue to be added to the final transaction
UniValue spends(UniValue::VOBJ);
UniValue received(UniValue::VOBJ);
UniValue sends(UniValue::VOBJ);
UniValue tx(UniValue::VOBJ);
//Begin Compiling the Decrypted Transaction
tx.push_back(Pair("txid", wtx.GetHash().ToString()));
if(fZdebug)
fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
if (wtx.IsCoinBase())
{
tx.push_back(Pair("coinbase", true));
if (wtx.GetDepthInMainChain() < 1)
tx.push_back(Pair("category", "orphan"));
else if (wtx.GetBlocksToMaturity() > 0)
tx.push_back(Pair("category", "immature"));
else
tx.push_back(Pair("category", "generate"));
} else {
tx.push_back(Pair("coinbase", false));
tx.push_back(Pair("category", "standard"));
}
tx.push_back(Pair("blockhash", wtx.hashBlock.GetHex()));
tx.push_back(Pair("blockindex", wtx.nIndex));
int confirms = wtx.GetDepthInMainChain();
if(confirms > 0)
{
tx.push_back(Pair("blocktime", mapBlockIndex[wtx.hashBlock]->GetBlockTime()));
} else {
tx.push_back(Pair("blocktime", 0));
}
tx.push_back(Pair("expiryheight", (int64_t)wtx.nExpiryHeight));
tx.push_back(Pair("confirmations", confirms));
tx.push_back(Pair("time", wtx.GetTxTime()));
tx.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
//Wallet Conflicts
UniValue conflicts(UniValue::VARR);
BOOST_FOREACH(const uint256& conflict, wtx.GetConflicts())
conflicts.push_back(conflict.GetHex());
tx.push_back(Pair("walletconflicts", conflicts));
// Return Type used to determine what is included in the transaction
// 0 Spends, Received and spends
// 1 Sends
// 2 Received
// 3 Spends
// TODO - Setup Enum, maybe...
// Add Spends retrieved earlier
if (returnType != 2) {
Legacy_zsTxSpendsToJSON(wtx, spends, totalSpends, filteredSpends, strAddress, fBool);
if ((!fBool || filteredSpends != 0) && (returnType == 0 || returnType == 1)) {
tx.push_back(Pair("spends",spends));
}
}
// Get Received
if (returnType == 0 || returnType == 2) {
Legacy_zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool);
if (!fBool || totalReceived != 0) {
tx.push_back(Pair("received",received));
}
}
// Get Sends
if (returnType == 0 || returnType == 3) {
//Only include sends if there are spends that belong to the wallet.
if (totalSpends != 0 || fBool) {
Legacy_zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool);
}
if (!fBool || totalSends != 0) {
tx.push_back(Pair("sends",sends));
}
}
if ((returnType == 0 && (!fBool || filteredSpends != 0 || totalReceived != 0 || totalSends != 0))
|| (returnType == 1 && (!fBool || filteredSpends != 0))
|| (returnType == 2 && (!fBool || totalReceived != 0))
|| (returnType == 3 && (!fBool || totalSends != 0))) {
ret.push_back(tx);
}
}
namespace {
std::array<unsigned char, HUSH_MEMO_SIZE> Memo(const std::string& text)
{
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
memo.fill(0);
if (text == "empty") // the "no memo" encoding: 0xF6 then zeros
memo[0] = 0xF6;
else if (text == "binary") // starts below 0xF5 but isn't UTF-8
memo[0] = 0xC3, memo[1] = 0x28;
else
memcpy(memo.data(), text.data(), text.size());
return memo;
}
// A real encrypted Sapling output, and when ovk is given, an outCiphertext the sender can decrypt with it
// (the wallet never checks proofs or value commitments).
OutputDescription Output(const SaplingPaymentAddress& to, uint64_t value, const std::string& memo,
const boost::optional<uint256>& ovk = boost::none)
{
SaplingNote note(to, value);
auto enc = SaplingNotePlaintext(note, Memo(memo)).encrypt(note.pk_d);
assert(static_cast<bool>(enc));
OutputDescription od;
od.cv = GetRandHash();
od.cm = note.cm().get();
od.ephemeralKey = enc->second.get_epk();
od.encCiphertext = enc->first;
if (ovk) {
SaplingOutgoingPlaintext out(note.pk_d, enc->second.get_esk());
od.outCiphertext = out.encrypt(*ovk, od.cv, od.cm, enc->second);
} else {
GetRandBytes(od.outCiphertext.begin(), od.outCiphertext.size());
}
return od;
}
CTransaction SaplingTx(const std::vector<OutputDescription>& outputs, const std::vector<uint256>& spentNullifiers,
const std::vector<COutPoint>& vin = {})
{
CMutableTransaction mtx;
mtx.fOverwintered = true;
mtx.nVersion = SAPLING_TX_VERSION;
mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
mtx.vShieldedOutput = outputs;
for (const uint256& nf : spentNullifiers) {
SpendDescription sd;
sd.nullifier = nf;
mtx.vShieldedSpend.push_back(sd);
}
for (const COutPoint& prev : vin)
mtx.vin.push_back(CTxIn(prev));
return CTransaction(mtx);
}
SaplingExtendedSpendingKey Key(uint32_t account)
{
std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x6b);
HDSeed seed(raw);
return SaplingExtendedSpendingKey::Master(seed).Derive(account | ZIP32_HARDENED_KEY_LIMIT);
}
}
// A wallet whose txs exercise every branch of the history helpers: notes received on default and diversified
// addresses, by a viewing-key-only address and by a key imported afterwards; z->z sends with change and memos;
// a t->z shielding send; spends of our notes and of someone else's. pwalletMain points at it during the test.
class HistoryWallet : public CWallet {
public:
bool EncryptKeys(CKeyingMaterial& vMasterKeyIn) { return CCryptoKeyStore::EncryptKeys(vMasterKeyIn); }
bool Unlock(const CKeyingMaterial& vMasterKeyIn) { return CCryptoKeyStore::Unlock(vMasterKeyIn); }
};
class WalletHistory : public ::testing::Test {
protected:
HistoryWallet wallet;
CWallet* savedWallet = nullptr;
SaplingPaymentAddress paA, paA2, paB, paC, paD, paF;
CTxDestination taddr;
std::map<uint256, int> depthOf;
std::vector<uint256> txids;
CCoinsView coinsBase;
CCoinsViewCache coins{&coinsBase};
CCoinsViewCache* savedCoinsTip = nullptr;
CBlockIndex* savedTip = nullptr;
std::deque<CBlock> blocks;
std::deque<CBlockIndex> index;
std::vector<uint256> insertedBlocks;
void SetUp() override {
SelectParams(CBaseChainParams::MAIN);
savedWallet = pwalletMain;
pwalletMain = &wallet;
savedCoinsTip = pcoinsTip;
pcoinsTip = &coins;
LOCK2(cs_main, wallet.cs_wallet);
savedTip = chainActive.Tip();
std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x2d);
ASSERT_TRUE(wallet.SetHDSeed(HDSeed(raw)));
HDSeed seed;
ASSERT_TRUE(wallet.GetHDSeedForDerivation(seed));
const uint256 seedOvk = ovkForShieldingFromTaddr(seed);
auto skA = Key(0), skB = Key(1), skC = Key(2), skD = Key(3), skF = Key(9);
paA = skA.DefaultAddress();
diversifier_index_t j;
j.begin()[0] = 50;
paA2 = skA.ToXFVK().Address(j)->second;
paB = skB.DefaultAddress();
paC = skC.DefaultAddress();
paD = skD.DefaultAddress();
paF = skF.DefaultAddress();
const uint256 ovkA = skA.expsk.full_viewing_key().ovk, ovkC = skC.expsk.full_viewing_key().ovk;
ASSERT_TRUE(wallet.AddSaplingZKey(skA, paA));
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(skA.expsk.full_viewing_key().in_viewing_key(), paA2));
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(skB.expsk.full_viewing_key().in_viewing_key(), paB));
ASSERT_TRUE(wallet.AddSaplingZKey(skC, paC));
CKey tkey;
tkey.MakeNewKey(true);
ASSERT_TRUE(wallet.AddKeyPubKey(tkey, tkey.GetPubKey()));
taddr = tkey.GetPubKey().GetID();
const uint256 nfA = uint256S("a1"), nfA2 = uint256S("a2"), nfC = uint256S("c1"), nfForeign = uint256S("ff");
// T0: transparent funds to our taddr.
CMutableTransaction t0;
t0.vin.push_back(CTxIn(COutPoint(GetRandHash(), 0)));
t0.vout.push_back(CTxOut(10 * COIN, GetScriptForDestination(taddr)));
AddTx(CTransaction(t0), 30);
// T1: received by A, A2, B (viewing key only), C and someone else; no spends.
uint256 t1 = AddTx(SaplingTx({Output(paA, 1000, "for A"), Output(paA2, 2000, "empty"), Output(paB, 3000, "for B"),
Output(paC, 4000, "binary"), Output(paF, 5000, "not ours")}, {}),
25, {{0, nfA}, {1, nfA2}, {3, nfC}});
(void)t1;
// T2: A sends to F with change back to A, spending A's note: outCiphertexts under A's ovk.
AddTx(SaplingTx({Output(paF, 600, "payment", ovkA), Output(paA, 399, "change", ovkA)}, {nfA}), 20);
// T3: t->z shielding of T0's output to A, outCiphertext under the seed's shielding ovk.
AddTx(SaplingTx({Output(paA, 9 * COIN, "empty", seedOvk)}, {}, {COutPoint(txids[0], 0)}), 15);
// T4: C spends its note: to A2 (another of our keys) and change to C.
AddTx(SaplingTx({Output(paA2, 1500, "to a diversified A", ovkC), Output(paC, 2499, "binary", ovkC)}, {nfC}), 10);
// T5: someone else's spend paying A.
AddTx(SaplingTx({Output(paA, 777, "from a stranger")}, {nfForeign}), 8);
// T6: received by D before D's key is in the wallet (an import without rescan), then mined.
AddTx(SaplingTx({Output(paD, 4242, "for D, found later")}, {}), 5);
// T8: spends one of our notes and one of someone else's (a coinjoin-like tx): missingSpendingKeys.
AddTx(SaplingTx({Output(paF, 100, "mixed", ovkA)}, {nfA, uint256S("fe")}), 3);
// T9: spends A2's note, but its one output's outCiphertext is under nobody's ovk we know: missingSaplingOVK.
AddTx(SaplingTx({Output(paF, 50, "unreadable send")}, {nfA2}), 2);
// T7: spends A2's note (A and A2 share A's key); never mined and not in the mempool (depth -1).
AddTx(SaplingTx({Output(paF, 2900, "", ovkA)}, {nfA2}), 0);
ASSERT_TRUE(wallet.AddSaplingZKey(skD, paD));
Mine();
}
void TearDown() override {
LOCK(cs_main);
chainActive.SetTip(savedTip);
for (const uint256& h : insertedBlocks)
mapBlockIndex.erase(h);
pcoinsTip = savedCoinsTip;
pwalletMain = savedWallet;
}
uint256 AddTx(const CTransaction& tx, int depth, const std::map<uint32_t, uint256>& nullifiers = {}) {
CWalletTx wtx(&wallet, tx);
mapSaplingNoteData_t found = wallet.FindMySaplingNotes(tx).first;
for (const auto& nf : nullifiers)
found.at(SaplingOutPoint(tx.GetHash(), nf.first)).nullifier = nf.second;
wtx.SetSaplingNoteData(found);
EXPECT_TRUE(wallet.AddToWallet(wtx, true, nullptr));
if (depth > 0)
depthOf[tx.GetHash()] = depth;
txids.push_back(tx.GetHash());
return tx.GetHash();
}
void Mine() {
const int tip = 40;
for (int h = 0; h <= tip; h++) {
blocks.emplace_back();
CBlock& block = blocks.back();
block.nTime = 1700000000 + 36 * h;
block.nNonce = uint256S(strprintf("%x", 0x900 + h));
if (h > 0)
block.hashPrevBlock = blocks[h - 1].GetHash();
for (const auto& d : depthOf)
if (tip - d.second + 1 == h)
block.vtx.push_back(wallet.mapWallet.at(d.first));
block.hashMerkleRoot = block.BuildMerkleTree();
index.emplace_back(block);
CBlockIndex& bi = index.back();
bi.SetHeight(h);
bi.pprev = h > 0 ? &index[h - 1] : nullptr;
auto inserted = mapBlockIndex.insert(std::make_pair(block.GetHash(), &bi));
ASSERT_TRUE(inserted.second);
insertedBlocks.push_back(block.GetHash());
bi.phashBlock = &inserted.first->first;
}
chainActive.SetTip(&index[tip]);
for (const auto& d : depthOf)
wallet.mapWallet.at(d.first).SetMerkleBranch(blocks[tip - d.second + 1]);
}
std::string NewJSON(const CWalletTx& wtx, const std::string& addr, bool fBool, int returnType) {
UniValue ret(UniValue::VARR);
zsWalletTxJSON(wtx, ret, addr, fBool, returnType, GetSaplingHistoryKeys());
return ret.write();
}
std::string LegacyJSON(const CWalletTx& wtx, const std::string& addr, bool fBool, int returnType) {
UniValue ret(UniValue::VARR);
Legacy_zsWalletTxJSON(wtx, ret, addr, fBool, returnType);
return ret.write();
}
void ExpectMatchesLegacy(bool checkCoverage = true);
};
// Every tx, every return type, unfiltered and filtered by each kind of address: byte-identical JSON. Run with
// the wallet's plaintext cache warm (as after the startup warm-up) and cold (every output trial-decrypted).
class WalletHistoryCold : public WalletHistory {};
TEST_F(WalletHistoryCold, MatchesLegacy) {
LOCK2(cs_main, wallet.cs_wallet);
for (auto& p : wallet.mapWallet)
for (auto& item : p.second.mapSaplingNoteData)
item.second.cachedPlaintext = boost::none;
ExpectMatchesLegacy();
}
TEST_F(WalletHistory, MatchesLegacy) {
LOCK2(cs_main, wallet.cs_wallet);
ExpectMatchesLegacy();
}
void WalletHistory::ExpectMatchesLegacy(bool checkCoverage) {
const std::vector<std::pair<std::string, bool>> filters = {
{"*", false}, {"*", true}, {EncodePaymentAddress(paA), true}, {EncodePaymentAddress(paA2), true},
{EncodePaymentAddress(paB), true}, {EncodePaymentAddress(paC), true}, {EncodePaymentAddress(paD), true},
{EncodePaymentAddress(paF), true}, {EncodeDestination(taddr), true},
};
std::string all;
for (const uint256& txid : txids) {
const CWalletTx& wtx = wallet.mapWallet.at(txid);
for (int returnType = 0; returnType <= 3; returnType++) {
for (const auto& f : filters) {
// New first: the legacy spends helper default-inserts into wallet maps.
std::string got = NewJSON(wtx, f.first, f.second, returnType);
std::string want = LegacyJSON(wtx, f.first, f.second, returnType);
EXPECT_EQ(want, got) << "tx " << txid.ToString() << " returnType " << returnType << " filter " << f.first << "/" << f.second;
all += got;
}
}
}
// The fixture really reaches the interesting branches.
if (!checkCoverage)
return;
for (const char* needle : {"\"change\":true", "\"memoStr\"", "\"saplingSpends\":[{", "\"transparentSpends\":[{",
"\"saplingSends\":[{", "\"saplingReceived\":[{", "\"missingSaplingOVK\":true",
"\"missingSpendingKeys\":true", "for D, found later"}) {
EXPECT_NE(std::string::npos, all.find(needle)) << needle;
}
}
// The same comparison on an encrypted wallet, locked (no spending keys or HD seed available: no shielded
// received or sent entries, as before) and then unlocked.
class WalletHistoryEncrypted : public WalletHistory {};
TEST_F(WalletHistoryEncrypted, MatchesLegacyLockedAndUnlocked) {
LOCK2(cs_main, wallet.cs_wallet);
CKeyingMaterial masterKey(32, 0x5e);
ASSERT_TRUE(wallet.EncryptKeys(masterKey));
ASSERT_TRUE(wallet.IsLocked());
ASSERT_TRUE(GetSaplingHistoryKeys().spendable.empty());
ExpectMatchesLegacy(false);
ASSERT_TRUE(wallet.Unlock(masterKey));
ASSERT_FALSE(GetSaplingHistoryKeys().spendable.empty());
ExpectMatchesLegacy();
}
// The history helpers only read the wallet. The old spends helper's operator[] lookups inserted an entry into
// mapSaplingNullifiersToNotes for every other wallet's nullifier they met, which made a later
// z_viewtransaction of that tx fail (map::at on a null outpoint).
TEST_F(WalletHistory, DoesNotModifyTheWallet) {
LOCK2(cs_main, wallet.cs_wallet);
const size_t nullifierEntries = wallet.mapSaplingNullifiersToNotes.size();
std::map<uint256, size_t> noteEntries;
for (const auto& p : wallet.mapWallet)
noteEntries[p.first] = p.second.mapSaplingNoteData.size();
for (const uint256& txid : txids)
for (int returnType = 0; returnType <= 3; returnType++)
NewJSON(wallet.mapWallet.at(txid), "*", false, returnType);
EXPECT_EQ(nullifierEntries, wallet.mapSaplingNullifiersToNotes.size());
EXPECT_EQ(0u, wallet.mapSaplingNullifiersToNotes.count(uint256S("ff"))) << "someone else's spend";
for (const auto& p : wallet.mapWallet)
EXPECT_EQ(noteEntries.at(p.first), p.second.mapSaplingNoteData.size());
// What the old helpers did, for the record.
for (const uint256& txid : txids)
LegacyJSON(wallet.mapWallet.at(txid), "*", false, 0);
EXPECT_EQ(1u, wallet.mapSaplingNullifiersToNotes.count(uint256S("ff")));
}
src/gtest/test_wallet_notecache.cpp
+1010
View File
File diff suppressed because it is too large Load Diff
src/gtest/test_wallet_notedata.cpp
+272
View File
@@ -0,0 +1,272 @@
// Copyright (c) 2016-2024 The Hush developers
// Distributed under the GPLv3 software license, see the accompanying
// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
#include <gtest/gtest.h>
#include <deque>
#include "chain.h"
#include "coins.h"
#include "main.h"
#include "primitives/block.h"
#include "primitives/transaction.h"
#include "random.h"
#include "uint256.h"
#include "wallet/wallet.h"
#include "zcash/Note.hpp"
#include "zcash/zip32.h"
namespace {
class NoteDataWallet : public CWallet {
public:
bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock = true) {
return CWallet::UpdatedNoteData(wtxIn, wtx, fSameBlock);
}
int VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessOnly) {
return CWallet::VerifyAndSetInitialWitness(pindex, witnessOnly);
}
};
libzcash::SaplingIncomingViewingKey TestIvk()
{
return libzcash::SaplingIncomingViewingKey(uint256S("0101010101010101010101010101010101010101010101010101010101010101"));
}
// A note as the wallet holds it once witnessed: nullifier set, witnesses cached at a height.
SaplingNoteData WitnessedNote(const SaplingMerkleTree& tree, int height, const uint256& nullifier)
{
SaplingNoteData nd(TestIvk(), nullifier);
nd.witnesses.push_front(tree.witness());
nd.witnessHeight = height;
return nd;
}
// A note as FindMySaplingNotes returns it during a replay or rescan: no nullifier, no witnesses.
SaplingNoteData FreshNote()
{
return SaplingNoteData(TestIvk());
}
}
// A replayed or rescanned tx must keep the witnesses AND the nullifier it already had, and must not
// report an update: dropping the nullifier made the next block clear and reseed every carried witness.
TEST(WalletNoteData, ReplayKeepsNullifierAndWitnesses) {
NoteDataWallet wallet;
CWalletTx wtx(&wallet, CTransaction());
uint256 txid = uint256S("aa");
SaplingOutPoint op0(txid, 0);
SaplingMerkleTree tree;
tree.append(uint256S("01"));
const uint256 nf = uint256S("f00d");
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 100, nf);
wtx.mapSaplingNoteData[op0].witnessRootValidated = true;
auto before = wtx.mapSaplingNoteData;
CWalletTx replay(&wallet, CTransaction());
replay.mapSaplingNoteData[op0] = FreshNote();
EXPECT_FALSE(wallet.UpdatedNoteData(replay, wtx));
ASSERT_EQ(1u, wtx.mapSaplingNoteData.size());
const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
ASSERT_TRUE(static_cast<bool>(nd.nullifier));
EXPECT_EQ(nf, *nd.nullifier);
EXPECT_EQ(100, nd.witnessHeight);
EXPECT_TRUE(nd.witnesses == before.at(op0).witnesses);
// ...but the kept witness is re-checked against the active chain once (in-memory flag only).
EXPECT_FALSE(nd.witnessRootValidated);
}
// A tx re-mined in a different block (reorg) has its notes at new tree positions: the old nullifier
// must not be carried, so VerifyAndSetInitialWitness reseeds the note at its new position.
TEST(WalletNoteData, MovedTxDropsNullifier) {
NoteDataWallet wallet;
CWalletTx wtx(&wallet, CTransaction());
SaplingOutPoint op0(uint256S("ee"), 0);
SaplingMerkleTree tree;
tree.append(uint256S("04"));
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 500, uint256S("cafe"));
CWalletTx moved(&wallet, CTransaction());
moved.mapSaplingNoteData[op0] = FreshNote();
EXPECT_TRUE(wallet.UpdatedNoteData(moved, wtx, false));
const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
EXPECT_FALSE(static_cast<bool>(nd.nullifier));
}
// When the incoming data really adds something (a note for a newly added key), the tx is updated,
// and the notes it already had keep their witnesses, witness height and nullifier together.
TEST(WalletNoteData, NewNoteUpdatesAndKeepsExistingNoteState) {
NoteDataWallet wallet;
CWalletTx wtx(&wallet, CTransaction());
uint256 txid = uint256S("bb");
SaplingOutPoint op0(txid, 0);
SaplingOutPoint op1(txid, 1);
SaplingMerkleTree tree;
tree.append(uint256S("02"));
const uint256 nf = uint256S("beef");
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 200, nf);
CWalletTx incoming(&wallet, CTransaction());
incoming.mapSaplingNoteData[op0] = FreshNote();
incoming.mapSaplingNoteData[op1] = FreshNote();
EXPECT_TRUE(wallet.UpdatedNoteData(incoming, wtx));
ASSERT_EQ(2u, wtx.mapSaplingNoteData.size());
const SaplingNoteData& nd0 = wtx.mapSaplingNoteData.at(op0);
ASSERT_TRUE(static_cast<bool>(nd0.nullifier));
EXPECT_EQ(nf, *nd0.nullifier);
EXPECT_EQ(200, nd0.witnessHeight);
EXPECT_EQ(1u, nd0.witnesses.size());
const SaplingNoteData& nd1 = wtx.mapSaplingNoteData.at(op1);
EXPECT_FALSE(static_cast<bool>(nd1.nullifier));
EXPECT_TRUE(nd1.witnesses.empty());
EXPECT_EQ(-1, nd1.witnessHeight);
}
// Nothing is carried for a note that has no witnesses (PART B2): its height and nullifier describe
// no witness, so they must not survive onto the incoming data.
TEST(WalletNoteData, UnwitnessedNoteCarriesNothing) {
NoteDataWallet wallet;
CWalletTx wtx(&wallet, CTransaction());
SaplingOutPoint op0(uint256S("cc"), 0);
SaplingNoteData stale(TestIvk(), uint256S("dead"));
stale.witnessHeight = 300;
wtx.mapSaplingNoteData[op0] = stale;
CWalletTx incoming(&wallet, CTransaction());
incoming.mapSaplingNoteData[op0] = FreshNote();
EXPECT_TRUE(wallet.UpdatedNoteData(incoming, wtx));
const SaplingNoteData& nd = wtx.mapSaplingNoteData.at(op0);
EXPECT_FALSE(static_cast<bool>(nd.nullifier));
EXPECT_EQ(-1, nd.witnessHeight);
EXPECT_TRUE(nd.witnesses.empty());
}
// Incoming data with no Sapling notes never touches the wallet's note data.
TEST(WalletNoteData, EmptyIncomingIsNoUpdate) {
NoteDataWallet wallet;
CWalletTx wtx(&wallet, CTransaction());
SaplingOutPoint op0(uint256S("dd"), 0);
SaplingMerkleTree tree;
tree.append(uint256S("03"));
wtx.mapSaplingNoteData[op0] = WitnessedNote(tree, 400, uint256S("abcd"));
CWalletTx incoming(&wallet, CTransaction());
EXPECT_FALSE(wallet.UpdatedNoteData(incoming, wtx));
EXPECT_EQ(1u, wtx.mapSaplingNoteData.size());
EXPECT_EQ(400, wtx.mapSaplingNoteData.at(op0).witnessHeight);
}
namespace {
// A real encrypted Sapling output to pa (the wallet never checks its proof or value commitment).
OutputDescription EncryptedOutput(const libzcash::SaplingPaymentAddress& pa, uint64_t value)
{
libzcash::SaplingNote note(pa, value);
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
memo.fill(0);
auto enc = libzcash::SaplingNotePlaintext(note, memo).encrypt(note.pk_d);
assert(static_cast<bool>(enc));
OutputDescription od;
od.cv = GetRandHash();
od.cm = note.cm().get();
od.ephemeralKey = enc->second.get_epk();
od.encCiphertext = enc->first;
return od;
}
// A three-block active chain (heights 0-2) whose block 1 holds the given txs, undone on destruction:
// chainActive, mapBlockIndex and pcoinsTip are process-wide and every gtest suite shares the process.
struct FakeActiveChain {
std::deque<CBlock> blocks;
std::deque<CBlockIndex> index;
CBlockIndex* savedTip;
CCoinsView coinsBase;
CCoinsViewCache coins{&coinsBase};
CCoinsViewCache* savedCoinsTip;
explicit FakeActiveChain(const std::vector<CTransaction>& inBlock1) {
LOCK(cs_main);
savedTip = chainActive.Tip();
savedCoinsTip = pcoinsTip;
pcoinsTip = &coins;
for (int h = 0; h <= 2; h++) {
blocks.emplace_back();
CBlock& block = blocks.back();
block.nTime = 1700000000 + h;
block.nNonce = uint256S(strprintf("%x", 0x700 + h));
if (h > 0)
block.hashPrevBlock = blocks[h - 1].GetHash();
if (h == 1)
block.vtx = inBlock1;
block.hashMerkleRoot = block.BuildMerkleTree();
index.emplace_back(block);
CBlockIndex& bi = index.back();
bi.SetHeight(h);
bi.pprev = h > 0 ? &index[h - 1] : nullptr;
bi.phashBlock = &mapBlockIndex.insert(std::make_pair(block.GetHash(), &bi)).first->first;
}
chainActive.SetTip(&index.back());
}
~FakeActiveChain() {
LOCK(cs_main);
chainActive.SetTip(savedTip);
for (const CBlock& block : blocks)
mapBlockIndex.erase(block.GetHash());
pcoinsTip = savedCoinsTip;
}
};
}
// A note found with an incoming viewing key only (z_importviewingkey) can't be spent and never gets
// a nullifier, because deriving one takes the full viewing key. VerifyAndSetInitialWitness must leave
// it alone. It used to clear it and reseed its witness from the block on disk on every call, then
// abort on the missing nullifier the first time the note was confirmed. Here there are no block
// files, so the old reseed path throws.
TEST(WalletNoteData, ViewingKeyOnlyNoteIsNotWitnessed) {
NoteDataWallet wallet;
std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x3c);
HDSeed seed(raw);
auto sk = libzcash::SaplingExtendedSpendingKey::Master(seed);
auto pa = sk.DefaultAddress();
LOCK2(cs_main, wallet.cs_wallet);
ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(sk.expsk.full_viewing_key().in_viewing_key(), pa));
ASSERT_FALSE(wallet.HaveSaplingFullViewingKey(sk.expsk.full_viewing_key().in_viewing_key()));
CMutableTransaction mtx;
mtx.fOverwintered = true;
mtx.nVersion = SAPLING_TX_VERSION;
mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
mtx.vShieldedOutput.push_back(EncryptedOutput(pa, 5000));
CTransaction tx(mtx);
CWalletTx wtx(&wallet, tx);
mapSaplingNoteData_t found = wallet.FindMySaplingNotes(tx).first;
ASSERT_EQ(1u, found.size()) << "the viewing key finds the note";
wtx.SetSaplingNoteData(found);
ASSERT_TRUE(wallet.AddToWallet(wtx, true, nullptr));
FakeActiveChain chain({tx});
CWalletTx& stored = wallet.mapWallet.at(tx.GetHash());
stored.SetMerkleBranch(chain.blocks[1]);
ASSERT_EQ(2, stored.GetDepthInMainChain());
int minHeight = -1;
ASSERT_NO_THROW(minHeight = wallet.VerifyAndSetInitialWitness(chainActive.Tip(), false));
EXPECT_EQ(chainActive.Height(), minHeight) << "it must not pull the witness rebuild down to its height";
const SaplingNoteData& nd = stored.mapSaplingNoteData.at(SaplingOutPoint(tx.GetHash(), 0));
EXPECT_TRUE(nd.witnesses.empty());
EXPECT_FALSE(static_cast<bool>(nd.nullifier));
EXPECT_EQ(-1, nd.witnessHeight);
}
src/init.cpp
+58 -1
View File
@@ -513,6 +513,10 @@ std::string HelpMessage(HelpMessageMode mode)
// If this is used incorrectly (-rescanheight too large), then the local wallet may attempt to spend funds which it does not have witness data about
// which will cause a "missing inputs" error when added to the mempool. Rescanning from correct height will fix this.
strUsage += HelpMessageOpt("-keepnotewitnesscache", _("Keep partial Sapling Note Witness cache. Must be used with -rescanheight to find missing cache items."));
if (showDebug)
strUsage += HelpMessageOpt("-notecachewarmup", "Decrypt all wallet notes into memory at startup, so the first balance call after a restart doesn't (default: 1)");
if (showDebug)
strUsage += HelpMessageOpt("-checknotecache", "On every balance or note-listing call, decrypt each cached note it considers again and log and repair any difference from the cache (slow; default: 0)");
strUsage += HelpMessageOpt("-rescan", _("Rescan the block chain for missing wallet transactions") + " " + _("on startup"));
strUsage += HelpMessageOpt("-rescanheight", _("Rescan from specified height when rescan=1 on startup"));
strUsage += HelpMessageOpt("-salvagewallet", _("Attempt to recover private keys from a corrupt wallet.dat") + " " + _("on startup"));
@@ -1016,7 +1020,8 @@ extern int32_t HUSH_REWIND;
// --- Adaptive coins-cache sizing -------------------------------------------------------------
// The in-memory UTXO/coins cache (nCoinCacheUsage) is the biggest lever on IBD speed: a bigger
// cache means far fewer chainstate flushes to disk. We size it to use most of RAM, but a scheduled
// cache means fewer size-triggered chainstate flushes to disk (the chainstate is also flushed at
// least every DATABASE_FLUSH_INTERVAL / DATABASE_FLUSH_BLOCKS, see main.h). We size it to use most of RAM, but a scheduled
// background task (AdjustCoinCacheForMemoryPressure, registered in AppInit2) shrinks the target when
// free system memory runs low — e.g. the user opens other apps — and grows it back when memory frees
// up, always leaving a reserve free for the rest of the system. The existing per-block flush
@@ -2229,6 +2234,9 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
// ********************************************************* Step 8: load wallet
#ifdef ENABLE_WALLET
// Set when the startup rescan below is interrupted: the wallet's state then ends at the scan's
// own checkpoint, not at the tip the notifier starts from.
bool fWalletScanIncomplete = false;
if (fDisableWallet) {
pwalletMain = NULL;
LogPrintf("Wallet disabled!\n");
@@ -2500,6 +2508,14 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
}
}
// Sweep and consolidation pick notes with GetFilteredNotes at minDepth > 1. Older versions
// located each note's block by re-reading its tx through the txindex, so on a -txindex=0 node
// every fully shielded tx was skipped and these never moved those notes. Now they will.
if (!fTxIndex && (pwalletMain->fSweepEnabled || pwalletMain->fSaplingConsolidationEnabled)) {
LogPrintf("%s: -txindex=0 with -zsweep/-consolidation: fully shielded notes, which versions before "
"v1.3.1 never saw on such nodes, are now included in sweeps and consolidations\n", __func__);
}
//Set Automatic Coinbase Shielding (default ON, conditional: self-guards
//on nodes where it cannot act - no owned coinbase, external mineraddress,
//or locked wallet). Closes the transparent-coinbase leak for miners.
@@ -2666,6 +2682,7 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
} else {
LogPrintf("Rescan did not complete; leaving the best-block locator at the scan's own "
"checkpoint so the remaining range is rescanned on the next start\n");
fWalletScanIncomplete = true;
}
// Restore wallet transaction metadata after -zapwallettxes=1
@@ -2693,6 +2710,15 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
}
}
}
// Decrypt the wallet's notes into the in-memory plaintext cache now, while RPC is still in warmup
// and before the wallet notification thread (started below) can want cs_wallet. Otherwise the
// first balance call after a restart decrypts them all under cs_main.
fCheckNoteCache = GetBoolArg("-checknotecache", false);
if (GetBoolArg("-notecachewarmup", true) && !ShutdownRequested()) {
uiInterface.InitMessage(_("Decrypting wallet notes..."));
pwalletMain->WarmSaplingNoteCache();
}
pwalletMain->SetBroadcastTransactions(GetBoolArg("-walletbroadcast", true));
} // (!fDisableWallet)
#else // ENABLE_WALLET
@@ -2736,6 +2762,21 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
{
LOCK(cs_main);
pindexLastTip = chainActive.Tip();
#ifdef ENABLE_WALLET
// The wallet's state reflects the block the notifier starts from, unless the startup
// rescan was interrupted: then it ends at the scan's checkpoint, and the hourly
// SetBestChain must keep stamping that so the next start rescans the rest.
if (pwalletMain) {
const CBlockIndex* pindexWallet = pindexLastTip;
if (fWalletScanIncomplete) {
CWalletDB walletdb(pwalletMain->strWalletFile);
CBlockLocator locator;
if (walletdb.ReadBestBlock(locator))
pindexWallet = FindForkInGlobalIndex(chainActive, locator);
}
pwalletMain->SetWalletTip(pindexWallet);
}
#endif
}
boost::function<void()> threadnotifywallets = boost::bind(&ThreadNotifyWallets, pindexLastTip);
threadGroup.create_thread(
@@ -2777,8 +2818,24 @@ bool AppInit2(boost::thread_group& threadGroup, CScheduler& scheduler)
uiInterface.InitMessage(_("Activating best chain..."));
// scan for better chains in the block chain database, that are not yet connected in the active best chain
CValidationState state;
const CBlockIndex* pindexBeforeActivate = NULL;
{
LOCK(cs_main);
pindexBeforeActivate = chainActive.Tip();
}
if ( !ActivateBestChain(true,state))
strErrors << "Failed to connect best block";
else {
// Persist a startup replay now: the wallet's witness rebuild runs next and can hold
// cs_main for a long time, and a node killed before any later flush would replay it all again.
bool fTipMoved;
{
LOCK(cs_main);
fTipMoved = chainActive.Tip() != pindexBeforeActivate;
}
if (fTipMoved)
FlushStateToDisk();
}
} else {
LogPrintf("HUSH_REWIND < 0\n");
}
src/main.cpp
+12 -2
View File
@@ -3738,6 +3738,7 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
static int64_t nLastWrite = 0;
static int64_t nLastFlush = 0;
static int64_t nLastSetChain = 0;
static int nLastFlushHeight = -1;
std::set<int> setFilesToPrune;
bool fFlushForPrune = false;
try {
@@ -3763,6 +3764,9 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
if (nLastSetChain == 0) {
nLastSetChain = nNow;
}
if (nLastFlushHeight < 0) {
nLastFlushHeight = chainActive.Height();
}
size_t cacheSize = pcoinsTip->DynamicMemoryUsage();
// The cache is large and close to the limit, but we have time now (not in the middle of a block processing).
bool fCacheLarge = mode == FLUSH_STATE_PERIODIC && cacheSize * (10.0/9) > nCoinCacheUsage;
@@ -3772,8 +3776,10 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
bool fPeriodicWrite = mode == FLUSH_STATE_PERIODIC && nNow > nLastWrite + (int64_t)DATABASE_WRITE_INTERVAL * 1000000;
// It's been very long since we flushed the cache. Do this infrequently, to optimize cache usage.
bool fPeriodicFlush = mode == FLUSH_STATE_PERIODIC && nNow > nLastFlush + (int64_t)DATABASE_FLUSH_INTERVAL * 1000000;
// Many blocks connected since the last flush, e.g. a long replay inside one ActivateBestChain call.
bool fBlockCountFlush = (mode == FLUSH_STATE_IF_NEEDED || mode == FLUSH_STATE_PERIODIC) && chainActive.Height() - nLastFlushHeight >= DATABASE_FLUSH_BLOCKS;
// Combine all conditions that result in a full cache flush.
bool fDoFullFlush = (mode == FLUSH_STATE_ALWAYS) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune;
bool fDoFullFlush = (mode == FLUSH_STATE_ALWAYS) || fCacheLarge || fCacheCritical || fPeriodicFlush || fFlushForPrune || fBlockCountFlush;
// Write blocks and block index to disk.
if (fDoFullFlush || fPeriodicWrite) {
// Depend on nMinDiskSpace to ensure we can write block index
@@ -3827,6 +3833,7 @@ bool static FlushStateToDisk(CValidationState &state, FlushStateMode mode) {
if (!pcoinsTip->Flush())
return AbortNode(state, "Failed to write to coin database");
nLastFlush = nNow;
nLastFlushHeight = chainActive.Height();
}
if ((mode == FLUSH_STATE_ALWAYS || mode == FLUSH_STATE_PERIODIC) && nNow > nLastSetChain + (int64_t)DATABASE_WRITE_INTERVAL * 1000000) {
// Update best block in wallet (so we can detect restored wallets).
@@ -5915,7 +5922,10 @@ FILE* OpenDiskFile(const CDiskBlockPos &pos, const char *prefix, bool fReadOnly)
return NULL;
boost::filesystem::path path = GetBlockPosFilename(pos, prefix);
boost::filesystem::create_directories(path.parent_path());
FILE* file = fopen(path.string().c_str(), "rb+");
// Read-only opens get a read-only FILE (every fReadOnly caller only reads). With "rb+", a write through a
// reused fd number landed here: when the wolfSSL CloseSocket macro (see net.h) closed the wallet's BDB fd
// behind its back, BDB's next page write went to whatever reopened that number -- a block file opened here.
FILE* file = fopen(path.string().c_str(), fReadOnly ? "rb" : "rb+");
if (!file && !fReadOnly)
file = fopen(path.string().c_str(), "wb+");
if (!file) {
src/main.h
+9 -2
View File
@@ -143,8 +143,15 @@ static const unsigned int MAX_LOCATOR_SZ = 101;
static unsigned int BLOCK_DOWNLOAD_WINDOW = 1024;
/** Time to wait (in seconds) between writing blocks/block index to disk. */
static const unsigned int DATABASE_WRITE_INTERVAL = 60 * 60;
/** Time to wait (in seconds) between flushing chainstate to disk. */
static const unsigned int DATABASE_FLUSH_INTERVAL = 24 * 60 * 60;
/** Time to wait (in seconds) between flushing chainstate to disk. Upstream uses 24h to keep a large
* coins cache warm, but DragonX's cache stays a few MiB (private chain, few transparent outputs), so
* the size triggers never fire and this timer is all that persists the chainstate while a node runs:
* a node that never reached 24h uptime or a clean shutdown replayed weeks of blocks on every start. */
static const unsigned int DATABASE_FLUSH_INTERVAL = 60 * 60;
/** Also flush chainstate once this many blocks were connected since the last flush. A long
* ActivateBestChain (startup replay, catching up after downtime) only flushes IF_NEEDED until it
* returns, so without this a node killed part-way through loses the whole replay. */
static const int DATABASE_FLUSH_BLOCKS = 5000;
/** Maximum length of reject messages. */
static const unsigned int MAX_REJECT_MESSAGE_LENGTH = 111;
static const int64_t DEFAULT_MAX_TIP_AGE = 24 * 60 * 60;
src/net.cpp
+7
View File
@@ -45,6 +45,13 @@
#include <wolfssl/options.h>
#include <wolfssl/ssl.h>
#include <hush/tlsmanager.cpp>
// CloseSocket must be util/sock.cpp's CloseSocket(SOCKET&), which invalidates the handle it closes. If
// wolfSSL's `#define CloseSocket(s) close(s)` survived into this file (see net.h), each close here would leave
// its stale fd number behind to be closed again after another thread reused it.
#ifdef CloseSocket
#error "CloseSocket is a macro here (wolfssl/wolfio.h): it would bypass CloseSocket(SOCKET&) in util/sock.cpp"
#endif
using namespace hush;
extern int32_t HUSH_TESTNODE;
extern uint8_t ASSETCHAINS_CLEARNET;
src/net.h
+8
View File
@@ -57,6 +57,14 @@
#define byte unsigned char
#include <wolfssl/ssl.h>
#undef byte
// wolfssl/wolfio.h (pulled in by ssl.h) does `#ifndef CloseSocket / #define CloseSocket(s) close(s)`. Left
// defined, it turns every CloseSocket(hSocket) after this point -- all 15 calls in net.cpp, including the ones in
// CloseSocketDisconnect and ~CNode -- into a bare close() that never sets hSocket = INVALID_SOCKET, so util/sock.cpp's
// CloseSocket(SOCKET&) is never called. Every P2P disconnect then closed its fd number again later (a repeated
// CloseSocketDisconnect, then ~CNode), by which time another thread often owned that number: wallet.dat or a
// database/log.* fd (BDB then wrote wallet pages into whatever got the number next, e.g. a blocks/*.dat file
// opened "rb+"), a block file, or an HTTP RPC socket. wolfio.h is include-guarded, so it can't come back.
#undef CloseSocket
class CAddrMan;
class CBlockIndex;
src/rpc/server.cpp
+26
View File
@@ -26,6 +26,7 @@
#include "util.h"
#include "util/strencodings.h"
#include "asyncrpcqueue.h"
#include <chrono>
#include <memory>
#include <univalue.h>
#include <boost/bind.hpp>
@@ -640,6 +641,30 @@ std::string JSONRPCExecBatch(const UniValue& vReq)
return ret.write() + "\n";
}
namespace {
// Logs an RPC whose wall time reached 500ms, or every RPC under -debug=bench. That time includes
// waiting for cs_main / cs_wallet behind block connection or another RPC, which the callee's own
// timers can't see because they start once the locks are held. A long-poll getblocktemplate waits
// for a new block with cs_main released, so it is only logged under -debug=bench. The method is
// logged as method=<name>: a bare " <name> " token would match the GUI's " rescan " log heuristic.
class RPCCallTimer {
const std::string& strMethod;
const bool fLongPoll;
const std::chrono::steady_clock::time_point start;
public:
RPCCallTimer(const std::string& method, const UniValue& params)
: strMethod(method),
fLongPoll(method == "getblocktemplate" && params.size() > 0 && params[0].isObject() &&
!find_value(params[0], "longpollid").isNull()),
start(std::chrono::steady_clock::now()) {}
~RPCCallTimer() {
const int64_t nMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - start).count();
if ((nMs >= 500 && !fLongPoll) || LogAcceptCategory("bench"))
LogPrintf("RPC method=%s took %dms\n", strMethod, nMs);
}
};
}
UniValue CRPCTable::execute(const std::string &strMethod, const UniValue &params) const
{
// Find method
@@ -674,6 +699,7 @@ UniValue CRPCTable::execute(const std::string &strMethod, const UniValue &params
try
{
// Execute
RPCCallTimer timer(strMethod, params);
return pcmd->actor(params, false, CPubKey());
}
catch (const std::exception& e)
src/validationinterface.cpp
+53 -18
View File
@@ -86,6 +86,30 @@ struct CachedBlockData {
pindex(pindex), oldTrees(oldTrees), txConflicted(txConflicted) {}
};
bool NotifyWalletsOrShutdown(const std::function<void()>& notify)
{
// The wallet can throw here on a database failure, e.g. CWalletDB::ListAccountCreditDebit's "error scanning
// DB" after BDB lost its file. Escaping this thread, that std::terminate()d the node (TraceThread logs and
// rethrows) with nothing flushed. Stop cleanly instead: going on would leave the wallet behind the chain,
// missing the block it failed on.
try {
notify();
return true;
} catch (const boost::thread_interrupted&) {
throw;
} catch (const std::exception& e) {
PrintExceptionContinue(&e, "ThreadNotifyWallets()");
} catch (...) {
PrintExceptionContinue(NULL, "ThreadNotifyWallets()");
}
LogPrintf("*** %s\n", "Wallet failed to process a block notification; shutting down");
StartShutdown();
uiInterface.ThreadSafeMessageBox(
_("Error: A fatal internal error occurred, see debug.log for details"),
"", CClientUIInterface::MSG_ERROR);
return false;
}
void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
{
// If pindexLastTip == nullptr, the wallet is at genesis.
@@ -203,15 +227,21 @@ void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
_("Error: A fatal internal error occurred, see debug.log for details"),
"", CClientUIInterface::MSG_ERROR);
StartShutdown();
// Don't hand the wallet an empty block as if it had been processed.
fDisconnectIncomplete = true;
break;
}
// Let wallets know transactions went from 1-confirmed to
// 0-confirmed or conflicted:
for (const CTransaction &tx : block.vtx) {
SyncWithWallets(tx, NULL);
}
// Update cached incremental witnesses
GetMainSignals().ChainTip(pindexLastTip, &block, boost::none);
if (!NotifyWalletsOrShutdown([&] {
// Let wallets know transactions went from 1-confirmed to
// 0-confirmed or conflicted:
for (const CTransaction &tx : block.vtx) {
SyncWithWallets(tx, NULL);
}
// Update cached incremental witnesses
GetMainSignals().ChainTip(pindexLastTip, &block, boost::none);
}))
return;
// On to the next block!
pindexLastTip = pindexLastTip->pprev;
@@ -239,19 +269,24 @@ void ThreadNotifyWallets(CBlockIndex *pindexLastTip)
_("Error: A fatal internal error occurred, see debug.log for details"),
"", CClientUIInterface::MSG_ERROR);
StartShutdown();
// Don't hand the wallet an empty block as if it had been processed.
break;
}
// Tell wallet about transactions that went from mempool
// to conflicted:
for (const CTransaction &tx : blockData.txConflicted) {
SyncWithWallets(tx, NULL);
}
// ... and about transactions that got confirmed:
for (const CTransaction &tx : block.vtx) {
SyncWithWallets(tx, &block);
}
// Update cached incremental witnesses
GetMainSignals().ChainTip(blockData.pindex, &block, blockData.oldTrees);
if (!NotifyWalletsOrShutdown([&] {
// Tell wallet about transactions that went from mempool
// to conflicted:
for (const CTransaction &tx : blockData.txConflicted) {
SyncWithWallets(tx, NULL);
}
// ... and about transactions that got confirmed:
for (const CTransaction &tx : block.vtx) {
SyncWithWallets(tx, &block);
}
// Update cached incremental witnesses
GetMainSignals().ChainTip(blockData.pindex, &block, blockData.oldTrees);
}))
return;
// This block is done!
pindexLastTip = blockData.pindex;
src/validationinterface.h
+5
View File
@@ -8,6 +8,7 @@
#define HUSH_VALIDATIONINTERFACE_H
#include <boost/signals2/signal.hpp>
#include <functional>
#include "zcash/IncrementalMerkleTree.hpp"
class CBlock;
@@ -76,4 +77,8 @@ CMainSignals& GetMainSignals();
void ThreadNotifyWallets(CBlockIndex *pindexLastTip);
/** Runs one block's wallet notifications for ThreadNotifyWallets. Returns false, after logging the error and
* requesting a shutdown, if they threw (e.g. a wallet database failure); thread interruption propagates. */
bool NotifyWalletsOrShutdown(const std::function<void()>& notify);
#endif // HUSH_VALIDATIONINTERFACE_H
src/version.h
+7 -1
View File
@@ -22,7 +22,13 @@
// network protocol versioning
// DragonX 1.0.0 - bumped to separate from old HUSH/DragonX nodes with RandomX bug
static const int PROTOCOL_VERSION = 2000000;
// DragonX 1.3.0 - bumped for the consensus changes in this release (BLOCK_VALID_CONTEXT
// no longer overwrites the validity level, the dead CBOPRET coinbase
// price check is gone, and pow.cpp lost 157 lines). MIN_PEER_PROTO_VERSION
// is deliberately NOT raised: it currently equals this value, and raising
// it would disconnect every node still on v1.0.3 -- which is all users,
// since no release has ever been installable (unsigned archives).
static const int PROTOCOL_VERSION = 2000001;
//! initial proto version, to be increased after version/verack negotiation
static const int INIT_PROTO_VERSION = 209;
//! In this version, 'getheaders' was introduced.
src/wallet/crypter.cpp
+1 -1
View File
@@ -591,7 +591,7 @@ bool CCryptoKeyStore::GetSaplingSpendingKey(const libzcash::SaplingFullViewingKe
if (!IsCrypted())
return CBasicKeyStore::GetSaplingSpendingKey(fvk, skOut);
for (auto entry : mapCryptedSaplingSpendingKeys) {
for (const auto& entry : mapCryptedSaplingSpendingKeys) {
if (entry.first.fvk == fvk) {
const std::vector<unsigned char> &vchCryptedSecret = entry.second;
return DecryptSaplingSpendingKey(vMasterKey, vchCryptedSecret, entry.first, skOut);
src/wallet/crypter.h
+1 -1
View File
@@ -265,7 +265,7 @@ public:
LOCK(cs_SpendingKeyStore);
if (!IsCrypted())
return CBasicKeyStore::HaveSaplingSpendingKey(fvk);
for (auto entry : mapCryptedSaplingSpendingKeys) {
for (const auto& entry : mapCryptedSaplingSpendingKeys) {
if (entry.first.fvk == fvk) {
return true;
}
src/wallet/db.cpp
+32 -9
View File
@@ -61,6 +61,7 @@ void CDBEnv::Reset()
dbenv = new DbEnv(DB_CXX_NO_EXCEPTIONS);
fDbEnvInit = false;
fMockDb = false;
fKeepLogs = false;
}
CDBEnv::CDBEnv() : dbenv(NULL)
@@ -276,10 +277,12 @@ bool CDBEnv::Salvage(const std::string& strFile, bool fAggressive, std::vector<C
void CDBEnv::CheckpointLSN(const std::string& strFile)
{
dbenv->txn_checkpoint(0, 0, 0);
if (dbenv->txn_checkpoint(0, 0, 0) != 0)
fKeepLogs = true;
if (fMockDb)
return;
dbenv->lsn_reset(strFile.c_str(), 0);
if (dbenv->lsn_reset(strFile.c_str(), 0) != 0)
fKeepLogs = true;
}
@@ -498,10 +501,16 @@ void CDBEnv::Flush(bool fShutdown)
// Move log data to the dat file
CloseDb(strFile);
LogPrint("db", "CDBEnv::Flush: %s checkpoint\n", strFile);
dbenv->txn_checkpoint(0, 0, 0);
int ret = dbenv->txn_checkpoint(0, 0, 0);
if (ret != 0) {
LogPrintf("CDBEnv::Flush: %s checkpoint failed: %s\n", strFile, DbEnv::strerror(ret));
fKeepLogs = true;
}
LogPrint("db", "CDBEnv::Flush: %s detach\n", strFile);
if (!fMockDb)
dbenv->lsn_reset(strFile.c_str(), 0);
if (!fMockDb && (ret = dbenv->lsn_reset(strFile.c_str(), 0)) != 0) {
LogPrintf("CDBEnv::Flush: %s lsn_reset failed: %s\n", strFile, DbEnv::strerror(ret));
fKeepLogs = true;
}
LogPrint("db", "CDBEnv::Flush: %s closed\n", strFile);
mapFileUseCount.erase(mi++);
} else
@@ -511,10 +520,24 @@ void CDBEnv::Flush(bool fShutdown)
if (fShutdown) {
char** listp;
if (mapFileUseCount.empty()) {
dbenv->log_archive(&listp, DB_ARCH_REMOVE);
Close();
if (!fMockDb)
boost::filesystem::remove_all(boost::filesystem::path(strPath) / "database");
// After a failed flush, or once the environment has panicked (BDB0060, e.g. after a page
// write failed), wallet.dat can be missing committed changes that only the logs hold.
// Removing them would turn a wallet the next start recovers (DB_RECOVER) into one that
// silently lost them. If the logs are unusable, the next start moves database/ aside.
// (get_flags itself fails with DB_RUNRECOVERY once the environment has panicked.)
u_int32_t envFlags = 0;
if (dbenv->get_flags(&envFlags) != 0 || (envFlags & DB_PANIC_ENVIRONMENT))
fKeepLogs = true;
if (fKeepLogs && !fMockDb) {
LogPrintf("CDBEnv::Flush: the wallet database was not flushed cleanly; keeping %s for recovery at the next start\n",
(boost::filesystem::path(strPath) / "database").string());
Close();
} else {
dbenv->log_archive(&listp, DB_ARCH_REMOVE);
Close();
if (!fMockDb)
boost::filesystem::remove_all(boost::filesystem::path(strPath) / "database");
}
}
}
}
src/wallet/db.h
+4
View File
@@ -28,6 +28,7 @@
#include "sync.h"
#include "version.h"
#include <atomic>
#include <map>
#include <string>
#include <vector>
@@ -53,6 +54,9 @@ class CDBEnv
private:
bool fDbEnvInit;
bool fMockDb;
// A checkpoint or lsn_reset failed: the logs in database/ may hold the only copy of committed
// wallet changes, so the shutdown flush must leave them for DB_RECOVER at the next start.
std::atomic<bool> fKeepLogs;
// Don't change into boost::filesystem::path, as that can result in
// shutdown problems/crashes caused by a static initialized internal pointer.
std::string strPath;
src/wallet/rpchushwallet.cpp
+165 -60
View File
@@ -16,6 +16,109 @@ using namespace libzcash;
bool EnsureWalletIsAvailable(bool avoidException);
SaplingHistoryKeys GetSaplingHistoryKeys()
{
AssertLockHeld(pwalletMain->cs_wallet);
SaplingHistoryKeys keys;
std::set<libzcash::SaplingPaymentAddress> addresses;
pwalletMain->GetSaplingPaymentAddresses(addresses);
for (const auto& addr : addresses) {
libzcash::SaplingExtendedSpendingKey extsk;
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
const libzcash::SaplingFullViewingKey fvk = extsk.expsk.full_viewing_key();
SaplingHistoryKeys::SpendableAddress a;
a.addr = addr;
a.ivk = fvk.in_viewing_key();
a.ovk = fvk.ovk;
// Left zero when the wallet doesn't know addr, exactly as GetNullifiersForAddresses does.
pwalletMain->GetSaplingIncomingViewingKey(addr, a.walletIvk);
keys.spendable.push_back(a);
}
}
return keys;
}
bool SaplingHistoryKeys::IsNullifierOfAddress(const libzcash::SaplingIncomingViewingKey& addrIvk, const uint256& nullifier) const
{
AssertLockHeld(pwalletMain->cs_wallet);
if (!noteNullifiers) {
noteNullifiers = std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256>>();
for (const auto& p : pwalletMain->mapWallet) {
for (const auto& item : p.second.mapSaplingNoteData) {
if (item.second.nullifier)
noteNullifiers->insert(std::make_pair(item.second.ivk, *item.second.nullifier));
}
}
}
return noteNullifiers->count(std::make_pair(addrIvk, nullifier)) > 0;
}
namespace {
// One output's decryptions under incoming viewing keys, each key tried at most once. Two distinct ivks can't
// both decrypt an output: the cm check recomputes pk_d = [ivk]g_d, and ivks are canonical scalars below the
// order of g_d (ka_agree and ivk_to_pkd reject the rest), so a second one would need a NoteCommit collision.
// So once it is known which ivk decrypts the output (the wallet's note data records the one FindMySaplingNotes
// found, with its plaintext normally cached), no other ivk is tried.
class OutputDecryptions {
const OutputDescription& output;
const SaplingNoteData* nd;
std::map<libzcash::SaplingIncomingViewingKey, boost::optional<SaplingNoteCache>> tried;
boost::optional<SaplingNoteCache> Decrypt(const libzcash::SaplingIncomingViewingKey& ivk) const {
auto both = libzcash::SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
if (!both)
return boost::none;
return SaplingNoteCache{ both->second, both->first.memo() };
}
// The plaintext under the ivk the wallet recorded for this output, decrypting it if it isn't cached.
const boost::optional<SaplingNoteCache>& Recorded() {
auto it = tried.find(nd->ivk);
if (it == tried.end())
it = tried.insert(std::make_pair(nd->ivk, nd->cachedPlaintext ? nd->cachedPlaintext : Decrypt(nd->ivk))).first;
return it->second;
}
public:
OutputDecryptions(const CWalletTx& wtx, uint32_t n) : output(wtx.vShieldedOutput[n]), nd(nullptr) {
auto it = wtx.mapSaplingNoteData.find(SaplingOutPoint(wtx.GetHash(), n));
if (it != wtx.mapSaplingNoteData.end())
nd = &it->second;
}
const SaplingNoteCache* Get(const libzcash::SaplingIncomingViewingKey& ivk) {
auto it = tried.find(ivk);
if (it == tried.end()) {
boost::optional<SaplingNoteCache> result;
if (nd && nd->ivk == ivk) {
result = Recorded();
} else if (!(nd && Recorded())) {
// Not a wallet note, or one whose recorded ivk doesn't decrypt it: try this key.
result = Decrypt(ivk);
}
it = tried.insert(std::make_pair(ivk, result)).first;
}
return it->second ? &*it->second : nullptr;
}
};
// What an outCiphertext tells the sender holding ovk: the note's plaintext and the address it was sent to.
boost::optional<std::pair<libzcash::SaplingNotePlaintext, libzcash::SaplingPaymentAddress>> DecryptSent(
const OutputDescription& outputDesc, const uint256& ovk)
{
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
outputDesc.outCiphertext,ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
if (!opt)
return boost::none;
auto opt_unwrapped = opt.get();
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
if (!pt)
return boost::none;
auto pt_unwrapped = pt.get();
return std::make_pair(pt_unwrapped, libzcash::SaplingPaymentAddress(pt_unwrapped.d, opt_unwrapped.pk_d));
}
}
void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) {
LOCK2(cs_main, pwalletMain->cs_wallet);
@@ -87,18 +190,29 @@ void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpen
const SpendDescription& spendDesc = wtx.vShieldedSpend[i];
UniValue obj(UniValue::VOBJ);
SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier];
// Lookups by find(), not operator[]: that inserted an entry for every other wallet's nullifier it met
// (and could insert an empty note into a wallet tx).
if (pwalletMain->IsSaplingNullifierFromMe(spendDesc.nullifier)) {
const CWalletTx* parent = pwalletMain->GetWalletTx(pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash);
const OutputDescription& output = parent->vShieldedOutput[op.n];
auto nd = pwalletMain->mapWallet[pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash].mapSaplingNoteData[op];
auto pt = libzcash::SaplingNotePlaintext::decrypt(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
const SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes.at(spendDesc.nullifier);
const CWalletTx* parent = pwalletMain->GetWalletTx(op.hash);
auto ndIt = parent->mapSaplingNoteData.find(op);
// The spent note's plaintext, from the wallet's cache when it has it.
boost::optional<SaplingNoteCache> pt;
if (ndIt != parent->mapSaplingNoteData.end() && op.n < parent->vShieldedOutput.size()) {
const SaplingNoteData& nd = ndIt->second;
if (nd.cachedPlaintext) {
pt = nd.cachedPlaintext;
} else {
const OutputDescription& output = parent->vShieldedOutput[op.n];
auto both = libzcash::SaplingNotePlaintext::decrypt_with_note(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
if (both)
pt = SaplingNoteCache{ both->second, both->first.memo() };
}
}
if (pt) {
auto note = pt.get();
auto pa = nd.ivk.address(note.d);
auto address = pa.get();
const libzcash::SaplingNote& note = pt->note;
const libzcash::SaplingPaymentAddress address(note.d, note.pk_d);
shieldedSpendsSpentCount++;
totalSpends += CAmount(-note.value());
obj.push_back(Pair("address",EncodePaymentAddress(address)));
@@ -132,7 +246,7 @@ void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpen
}
void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress) {
void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys) {
if(fZdebug)
fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
@@ -188,22 +302,18 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
UniValue zsReceived(UniValue::VARR);
if (isZsAddress || !filterByAddress) {
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
UniValue obj(UniValue::VOBJ);
bool changeTx = false;
//Decrypt sapling incoming commitments using IVK
std::set<libzcash::SaplingPaymentAddress> addresses;
pwalletMain->GetSaplingPaymentAddresses(addresses);
for (auto addr : addresses) {
libzcash::SaplingExtendedSpendingKey extsk;
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext, extsk.expsk.full_viewing_key().in_viewing_key(), outputDesc.ephemeralKey, outputDesc.cm);
//Decrypt sapling incoming commitments using IVK: once per key, not once per address
OutputDecryptions decryptions(wtx, i);
for (const SaplingHistoryKeys::SpendableAddress& a : keys.spendable) {
const libzcash::SaplingPaymentAddress& addr = a.addr;
{
const SaplingNoteCache* pt = decryptions.Get(a.ivk);
if (pt) {
auto note = pt.get();
auto pt_unwrapped = pt.get();
auto memo = pt_unwrapped.memo();
const libzcash::SaplingNote& note = pt->note;
auto memo = pt->memo;
obj.push_back(Pair("address",EncodePaymentAddress(addr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value()))));
obj.push_back(Pair("memo", HexStr(memo)));
@@ -219,10 +329,8 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
//Check Change Status
if (wtx.vShieldedSpend.size()!=0) {
std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
nullifierSet = pwalletMain->GetNullifiersForAddresses({addr});
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
if (nullifierSet.count(std::make_pair(addr, spendDesc.nullifier))) {
if (keys.IsNullifierOfAddress(a.walletIvk, spendDesc.nullifier)) {
changeTx = true;
}
}
@@ -243,7 +351,7 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
}
void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress) {
void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys) {
LOCK2(cs_main, pwalletMain->cs_wallet);
@@ -296,6 +404,13 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
//All Shielded Sends in the transaction
UniValue zsSends(UniValue::VARR);
if (isZsAddress || !filterByAddress) {
//The t to z shielding ovk comes from the HD seed: derive it once per tx, not once per output
boost::optional<uint256> seedOvk;
if (wtx.vShieldedSpend.size()==0 && wtx.vShieldedOutput.size()!=0) {
HDSeed seed;
if (pwalletMain->GetHDSeedForDerivation(seed))
seedOvk = ovkForShieldingFromTaddr(seed);
}
for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
shieldedOutputCount++;
@@ -304,20 +419,14 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
//Decrypt sapling outgoing t to z transaction using HDseed
if (wtx.vShieldedSpend.size()==0) {
HDSeed seed;
if (pwalletMain->GetHDSeedForDerivation(seed)) {
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
outputDesc.outCiphertext,ovkForShieldingFromTaddr(seed),outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
if (seedOvk) {
{
auto sent = DecryptSent(outputDesc, *seedOvk);
if (opt) {
auto opt_unwrapped = opt.get();
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
if (pt) {
if (sent) {
shieldedOutputDecryptedCount++;
auto pt_unwrapped = pt.get();
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
const libzcash::SaplingNotePlaintext& pt_unwrapped = sent->first;
const libzcash::SaplingPaymentAddress& sentAddr = sent->second;
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value()))));
obj.push_back(Pair("shieldedOutputIndex",i));
@@ -330,26 +439,22 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
}
}
//attempt Decryption of Outgoing Sapling using wallet extended spending keys
//attempt Decryption of Outgoing Sapling using wallet extended spending keys: once per ovk, not once per address
} else {
std::set<libzcash::SaplingPaymentAddress> addresses;
pwalletMain->GetSaplingPaymentAddresses(addresses);
for (auto addr : addresses) {
libzcash::SaplingExtendedSpendingKey extsk;
if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
outputDesc.outCiphertext,extsk.expsk.full_viewing_key().ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
std::map<uint256, boost::optional<std::pair<libzcash::SaplingNotePlaintext, libzcash::SaplingPaymentAddress>>> sentByOvk;
for (const SaplingHistoryKeys::SpendableAddress& a : keys.spendable) {
{
auto tried = sentByOvk.find(a.ovk);
if (tried == sentByOvk.end())
tried = sentByOvk.insert(std::make_pair(a.ovk, DecryptSent(outputDesc, a.ovk))).first;
const auto& sent = tried->second;
if (opt) {
auto opt_unwrapped = opt.get();
auto pt = libzcash::SaplingNotePlaintext::decrypt(
outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
if (pt) {
auto pt_unwrapped = pt.get();
{
if (sent) {
const libzcash::SaplingNotePlaintext& pt_unwrapped = sent->first;
auto memo = pt_unwrapped.memo();
shieldedOutputDecryptedCount++;
libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
const libzcash::SaplingPaymentAddress& sentAddr = sent->second;
obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value()))));
obj.push_back(Pair("memo", HexStr(memo)));
@@ -365,10 +470,10 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
//Check Change Status
if (wtx.vShieldedSpend.size()!=0) {
std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
nullifierSet = pwalletMain->GetNullifiersForAddresses({sentAddr});
libzcash::SaplingIncomingViewingKey sentIvk;
pwalletMain->GetSaplingIncomingViewingKey(sentAddr, sentIvk);
BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
if (nullifierSet.count(std::make_pair(sentAddr, spendDesc.nullifier))) {
if (keys.IsNullifierOfAddress(sentIvk, spendDesc.nullifier)) {
changeTx = true;
}
}
@@ -397,7 +502,7 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
}
void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType) {
void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, const SaplingHistoryKeys& keys) {
LOCK2(cs_main, pwalletMain->cs_wallet);
@@ -469,7 +574,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
}
// Get Received
if (returnType == 0 || returnType == 2) {
zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool);
zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool, keys);
if (!fBool || totalReceived != 0) {
tx.push_back(Pair("received",received));
}
@@ -479,7 +584,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
if (returnType == 0 || returnType == 3) {
//Only include sends if there are spends that belong to the wallet.
if (totalSpends != 0 || fBool) {
zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool);
zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool, keys);
}
if (!fBool || totalSends != 0) {
tx.push_back(Pair("sends",sends));
src/wallet/rpchushwallet.h
+28 -3
View File
@@ -13,10 +13,35 @@ struct balancestruct {
CAmount immature;
};
/** The wallet's Sapling keys as the history helpers use them, gathered once per RPC call. The helpers used to
* look them up for every output and every address, re-deriving each full viewing key (an elliptic-curve
* operation), and walked every note in the wallet for each change check. Build it with pwalletMain->cs_wallet
* held, and keep holding it while the helpers run: that keeps the wallet's keys and notes fixed. The one thing
* it can't stop is the walletpassphrase timer relocking the wallet mid-call; the call then reports the keys as
* they were when it started, where the old per-output lookups switched over partway. */
struct SaplingHistoryKeys {
struct SpendableAddress {
libzcash::SaplingPaymentAddress addr;
libzcash::SaplingIncomingViewingKey ivk; //!< from its spending key: what received outputs are decrypted with
uint256 ovk; //!< from its spending key: what outCiphertexts are decrypted with
libzcash::SaplingIncomingViewingKey walletIvk; //!< GetSaplingIncomingViewingKey(addr), as the change check looked it up
};
//! Every wallet address whose spending key is available (never while the wallet is locked), in
//! GetSaplingPaymentAddresses order.
std::vector<SpendableAddress> spendable;
//! Whether GetNullifiersForAddresses({addr}) would contain (addr, nullifier), given addr's
//! GetSaplingIncomingViewingKey result (zero when the wallet doesn't know addr): nullifier belongs to a
//! wallet note received with that ivk. Walks the wallet's notes once, on first use.
bool IsNullifierOfAddress(const libzcash::SaplingIncomingViewingKey& addrIvk, const uint256& nullifier) const;
private:
mutable boost::optional<std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256>>> noteNullifiers;
};
SaplingHistoryKeys GetSaplingHistoryKeys();
void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress);
void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress);
void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress);
void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType);
void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys);
void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys);
void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, const SaplingHistoryKeys& keys);
#endif //HUSH_WALLET_RPCWALLET_H
src/wallet/rpcwallet.cpp
+28 -19
View File
@@ -3032,11 +3032,13 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
}
//Create Ordered List
map<int64_t,CWalletTx> orderedTxs;
// Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
// As before, the first tx inserted for an nOrderPos wins.
map<int64_t,const CWalletTx*> orderedTxs;
for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
const uint256& wtxid = it->first;
const CWalletTx& wtx = (*it).second;
orderedTxs.insert(std::pair<int64_t,CWalletTx>(wtx.nOrderPos, wtx));
orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
unsigned int txType = 0;
// 0 Unassigend
@@ -3228,10 +3230,11 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
}
const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
uint64_t t = GetTime();
for (map<int64_t,CWalletTx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it)
for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it)
{
const CWalletTx& wtx = (*it).second;
const CWalletTx& wtx = *(*it).second;
if (!CheckFinalTx(wtx))
continue;
@@ -3248,7 +3251,7 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
continue;
}
zsWalletTxJSON(wtx, trans, "*", false, 0);
zsWalletTxJSON(wtx, trans, "*", false, 0, historyKeys);
if (trans.size() >= nCount) break;
}
@@ -3515,17 +3518,20 @@ UniValue z_listreceivedaddress(const UniValue& params, bool fHelp,const CPubKey&
throw runtime_error("Filter must be greater that 0.");
//Created Ordered Transaction Map
map<int64_t,CWalletTx> orderedTxs;
// Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
// As before, the first tx inserted for an nOrderPos wins.
map<int64_t,const CWalletTx*> orderedTxs;
for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
const CWalletTx& wtx = (*it).second;
orderedTxs.insert(std::pair<int64_t,CWalletTx>(wtx.nOrderPos, wtx));
orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
}
const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
uint64_t t = GetTime();
//Reverse Iterate thru transactions
for (map<int64_t,CWalletTx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
const CWalletTx& wtx = (*it).second;
for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
const CWalletTx& wtx = *(*it).second;
bool includeTransaction = true;
//Excude transactions with less confirmations than required
@@ -3546,7 +3552,7 @@ UniValue z_listreceivedaddress(const UniValue& params, bool fHelp,const CPubKey&
}
if (includeTransaction) {
zsWalletTxJSON(wtx, ret, params[0].get_str() , true, 2);
zsWalletTxJSON(wtx, ret, params[0].get_str() , true, 2, historyKeys);
}
if (ret.size() > nCount) break;
@@ -3698,17 +3704,20 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
throw runtime_error("Filter must be greater that 0.");
//Created Ordered Transaction Map
map<int64_t,CWalletTx> orderedTxs;
// Pointers into mapWallet, not copies (a CWalletTx copy deep-copies every note's witness list).
// As before, the first tx inserted for an nOrderPos wins.
map<int64_t,const CWalletTx*> orderedTxs;
for (map<uint256, CWalletTx>::iterator it = pwalletMain->mapWallet.begin(); it != pwalletMain->mapWallet.end(); ++it) {
const CWalletTx& wtx = (*it).second;
orderedTxs.insert(std::pair<int64_t,CWalletTx>(wtx.nOrderPos, wtx));
orderedTxs.insert(std::make_pair(wtx.nOrderPos, &wtx));
}
const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
uint64_t t = GetTime();
//Reverse Iterate thru transactions
for (map<int64_t,CWalletTx>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
const CWalletTx& wtx = (*it).second;
for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
const CWalletTx& wtx = *(*it).second;
if (!CheckFinalTx(wtx))
continue;
@@ -3731,7 +3740,7 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
if (nFilterType == 2 && wtx.GetDepthInMainChain() > nFilter)
continue;
zsWalletTxJSON(wtx, ret, "*", false, 0);
zsWalletTxJSON(wtx, ret, "*", false, 0, historyKeys);
if (ret.size() >= nCount) break;
@@ -4309,7 +4318,7 @@ UniValue z_listunspent(const UniValue& params, bool fHelp, const CPubKey& mypk)
for (auto & entry : saplingEntries) {
UniValue obj(UniValue::VOBJ);
int nHeight = tx_height(entry.op.hash);
int nHeight = HeightFromDepth(entry.confirmations);
int dpowconfs = hush_dpowconfs(nHeight, entry.confirmations);
// Only return notarized results when minconf>1
@@ -4642,7 +4651,7 @@ UniValue z_listreceivedbyaddress(const UniValue& params, bool fHelp, const CPubK
if (boost::get<libzcash::SaplingPaymentAddress>(&zaddr) != nullptr) {
for (SaplingNoteEntry & entry : saplingEntries) {
UniValue obj(UniValue::VOBJ);
int nHeight = tx_height(entry.op.hash);
int nHeight = HeightFromDepth(entry.confirmations);
int dpowconfs = hush_dpowconfs(nHeight, entry.confirmations);
// Only return notarized results when minconf>1
if (nMinDepth > 1 && dpowconfs == 1)
@@ -4662,7 +4671,7 @@ UniValue z_listreceivedbyaddress(const UniValue& params, bool fHelp, const CPubK
}
obj.push_back(Pair("outindex", (int)entry.op.n));
obj.push_back(Pair("rawconfirmations", entry.confirmations));
auto wtx = pwalletMain->mapWallet.at(entry.op.hash); //.ToString());
const CWalletTx& wtx = pwalletMain->mapWallet.at(entry.op.hash);
obj.push_back(Pair("time", wtx.GetTxTime()));
obj.push_back(Pair("confirmations", dpowconfs));
@@ -5376,7 +5385,7 @@ UniValue z_sendmany(const UniValue& params, bool fHelp, const CPubKey& mypk)
// sort notes from largest to smallest, which means
// we will spend the largest first
std::sort(saplingEntries.begin(), saplingEntries.end(),
[](SaplingNoteEntry i, SaplingNoteEntry j) -> bool {
[](const SaplingNoteEntry& i, const SaplingNoteEntry& j) -> bool {
return i.note.value() > j.note.value();
});
src/wallet/wallet.cpp
+408 -93
View File
@@ -43,7 +43,9 @@
#include "wallet/asyncrpcoperation_autoshieldcoinbase.h"
#include <random>
#include <limits>
#include <system_error>
#include <thread>
#include <chrono>
#include <atomic>
#include <mutex>
#include "zcash/zip32.h"
@@ -75,8 +77,8 @@ bool fPayAtLeastCustomFee = true;
CBlockIndex *hush_chainactive(int32_t height);
extern int32_t HUSH_LOADINGBLOCKS;
int32_t hush_dpowconfs(int32_t height,int32_t numconfs);
int tx_height( const uint256 &hash );
bool fTxDeleteEnabled = false;
bool fCheckNoteCache = false;
bool fTxConflictDeleteEnabled = false;
int fDeleteInterval = DEFAULT_TX_DELETE_INTERVAL;
unsigned int fDeleteTransactionsAfterNBlocks = DEFAULT_TX_RETENTION_BLOCKS;
@@ -575,6 +577,23 @@ void CWallet::ChainTip(const CBlockIndex *pindex,
DecrementNoteWitnesses(pindex);
UpdateNullifierNoteMapForBlock(pblock);
}
// Track the block the wallet's state reflects, advancing only along blocks delivered in order:
// a gap (the startup rescan was interrupted, so the wallet sits at its checkpoint behind the
// notifier) pins the pointer, and SetBestChain keeps stamping the checkpoint.
LOCK(cs_wallet);
if (added) {
if (pindexWalletTip == nullptr || pindex->pprev == pindexWalletTip)
pindexWalletTip = pindex;
} else if (pindex == pindexWalletTip) {
pindexWalletTip = pindex->pprev;
}
}
void CWallet::SetWalletTip(const CBlockIndex* pindex)
{
LOCK(cs_wallet);
pindexWalletTip = pindex;
}
void CWallet::RunSaplingSweep(int blockHeight) {
@@ -801,10 +820,22 @@ bool CWallet::CommitAutomatedTx(const CTransaction& tx) {
void CWallet::SetBestChain(const CBlockLocator& loc)
{
// FlushStateToDisk passes the chain tip, but ThreadNotifyWallets delivers blocks to the wallet
// asynchronously and can lag it. Stamping a locator ahead of what the wallet has processed makes
// the next start skip those blocks for good (no rescan, no notification): notes received in them
// go missing and spends in them go unseen. So stamp the block the wallet's state reflects (seeded
// by init before the notifier starts, then advanced by ChainTip). Init's own calls, made before
// it is seeded, pass the right locator themselves.
CBlockLocator locator = loc;
{
LOCK2(cs_main, cs_wallet);
if (pindexWalletTip != nullptr)
locator = chainActive.GetLocator(pindexWalletTip);
}
// Default ctor => fFlushOnClose=true => ~CDB runs a full BDB txn_checkpoint over the entire
// cache. Fine for the hourly/shutdown callers; ruinous inside a scan (see SetBestChainNoFlush).
CWalletDB walletdb(strWalletFile);
SetBestChainINTERNAL(walletdb, loc);
SetBestChainINTERNAL(walletdb, locator);
}
bool CWallet::SetBestChainNoFlush(const CBlockLocator& loc)
@@ -1232,9 +1263,11 @@ void ClearSingleNoteWitnessCache(NoteData* nd)
nd->witnessRootValidated = false;
}
int CWallet::SaplingWitnessMinimumHeight(const uint256& nullifier, int nWitnessHeight, int nMinimumHeight)
// A note without a nullifier is never extended by BuildWitnessCache (its pending-note gate requires one),
// so it doesn't pull the rebuild's start height down.
int CWallet::SaplingWitnessMinimumHeight(const boost::optional<uint256>& nullifier, int nWitnessHeight, int nMinimumHeight)
{
if (GetSaplingSpendDepth(nullifier) <= WITNESS_CACHE_SIZE) {
if (nullifier && GetSaplingSpendDepth(*nullifier) <= WITNESS_CACHE_SIZE) {
nMinimumHeight = min(nWitnessHeight, nMinimumHeight);
}
return nMinimumHeight;
@@ -1266,6 +1299,14 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
uint256 blockRoot;
uint256 witnessRoot;
//A note found with an incoming viewing key only (z_importviewingkey) can't be spent, so it needs
//no witness, and it never gets a nullifier: deriving one takes the full viewing key, which
//UpdateSaplingNullifierNoteMapWithTx skips without. Seeding it here meant clearing and reseeding
//it from disk on every call (the no-nullifier rule below), then aborting on the missing nullifier
//the first time it was confirmed. If its spending key is imported later, the rule below seeds it.
if (!HaveSaplingFullViewingKey(nd->ivk))
continue;
if (!nd->nullifier)
::ClearSingleNoteWitnessCache(nd);
@@ -1277,7 +1318,7 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
//Skip Validation when witness root has been validated
if (nd->witnessRootValidated) {
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
continue;
}
@@ -1289,12 +1330,12 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
CBlockIndex* whIndex = chainActive[nd->witnessHeight];
if (whIndex == NULL) {
//witnessHeight strictly above the active chain (transient catch-up): cannot validate yet
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
continue;
}
if (nd->witnesses.front().root() == whIndex->hashFinalSaplingRoot) {
nd->witnessRootValidated = true;
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
continue;
}
//root mismatch on the active chain -> desynced; fall through to rebuild below
@@ -1306,7 +1347,7 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
blockRoot = pblockindex->hashFinalSaplingRoot;
if (witnessRoot == blockRoot) {
nd->witnessRootValidated = true;
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
continue;
}
}
@@ -1358,7 +1399,7 @@ int CWallet::VerifyAndSetInitialWitness(const CBlockIndex* pindex, bool witnessO
}
nd->witnessHeight = pblockindex->GetHeight();
UpdateSaplingNullifierNoteMapWithTx(wtxItem.second);
nMinimumHeight = SaplingWitnessMinimumHeight(*item.second.nullifier, nd->witnessHeight, nMinimumHeight);
nMinimumHeight = SaplingWitnessMinimumHeight(item.second.nullifier, nd->witnessHeight, nMinimumHeight);
}
}
}
@@ -1477,17 +1518,25 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
work.reserve(pending.size());
for (const PendingNote& p : pending) work.push_back(p.nd);
// Only a substantial bulk rebuild (e.g. a one-time post-upgrade repair) is worth parallelizing
// and logging; routine 1-block tip extension runs serially to avoid per-block thread-spawn
// overhead and log spam.
// Only a substantial bulk rebuild (e.g. a one-time post-upgrade repair) is logged and has its
// anchors checked; routine 1-block tip extension stays quiet to avoid per-block log spam.
const bool bulkRebuild = (rangeLen > 100);
// Parallelize whenever the extension itself is expensive, not only for a bulk range: a routine
// 1-block extension appends each of the block's commitments to every lagging note, so a large
// wallet and a busy block (13k notes x 20 commitments) is seconds of hashing under
// cs_main/cs_wallet. Spawning the threads costs ~1ms, so only small jobs stay serial.
uint64_t nCms = 0;
for (const std::vector<uint256>& cms : blockCms)
nCms += cms.size();
const bool fParallel = bulkRebuild || (uint64_t)work.size() * nCms >= 2000;
int nPar = (int)GetArg("-witnessbuildthreads", 0);
if (nPar <= 0) nPar = (int)std::thread::hardware_concurrency();
if (nPar <= 0) nPar = 1;
if (nPar > (int)work.size()) nPar = (int)work.size();
if (nPar < 1) nPar = 1;
if (!bulkRebuild) nPar = 1;
if (!fParallel) nPar = 1;
std::atomic<bool> failed(false);
std::mutex failMtx;
@@ -1499,14 +1548,27 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
// clone-every-block path for A/B verification.
bool fastRebuild = GetBoolArg("-witnessfastrebuild", true);
// A bulk rebuild reports how many notes failed to reach the canonical anchor. root() walks the
// full depth-32 path (~2.4ms), so the check runs here on the workers and never on the routine
// per-block path, where it used to hold cs_main/cs_wallet for ~30s per block on a ~13k-note
// wallet purely to feed a log line that path never prints.
const uint256 canonicalRoot = pindex->hashFinalSaplingRoot;
std::atomic<int> nUnrecoverable(0);
auto checkRoot = [&](const SaplingNoteData* nd) {
if (bulkRebuild && (nd->witnesses.empty() || nd->witnesses.front().root() != canonicalRoot))
nUnrecoverable++;
};
auto worker = [&](int tid) {
try {
for (size_t k = (size_t)tid; k < work.size(); k += (size_t)nPar) {
SaplingNoteData* nd = work[k];
int startH = nd->witnessHeight;
int nBlocks = tipHeight - startH;
if (nBlocks <= 0)
if (nBlocks <= 0) {
checkRoot(nd);
continue;
}
if (!fastRebuild || nBlocks <= CACHE) {
// Reference / shallow path: clone every block (preserves pre-existing older snapshots).
@@ -1547,6 +1609,7 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
nd->witnesses.pop_back();
nd->witnessHeight = tipHeight;
}
checkRoot(nd);
}
} catch (const std::exception& e) {
std::lock_guard<std::mutex> lk(failMtx);
@@ -1561,9 +1624,18 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
if (nPar <= 1) {
worker(0);
} else {
// If the OS refuses a thread (task limit, memory), run the remaining shards here instead of
// letting the exception unwind past joinable threads (std::terminate). Each shard still runs
// exactly once, so the result is unchanged.
std::vector<std::thread> threads;
threads.reserve(nPar);
for (int t = 0; t < nPar; t++) threads.emplace_back(worker, t);
int t = 0;
try {
for (; t < nPar; t++) threads.emplace_back(worker, t);
} catch (const std::system_error& e) {
LogPrintf("%s: could only start %d of %d witness threads (%s); running the rest inline\n", __func__, t, nPar, e.what());
}
for (int u = t; u < nPar; u++) worker(u);
for (std::thread& th : threads) th.join();
}
int64_t tExtend = GetTimeMicros() - e0;
@@ -1578,20 +1650,16 @@ void CWallet::BuildWitnessCache(const CBlockIndex* pindex, bool witnessOnly)
// is unrecoverable here: it is left flagged (the GetSaplingNoteWitnesses majority-anchor guard
// skips it for spends) and reported, rather than spun on indefinitely. witnessRootValidated is
// in-memory only, so a fresh validation pass still runs on each restart and after any reorg
// (DecrementNoteWitnesses clears it), keeping the heal self-correcting.
// (DecrementNoteWitnesses clears it), keeping the heal self-correcting. The unrecoverable count
// itself comes from checkRoot() on the workers, and only for a bulk rebuild.
if (!work.empty()) {
const uint256& canonicalRoot = pindex->hashFinalSaplingRoot;
int nUnrecoverable = 0;
for (SaplingNoteData* nd : work) {
if (nd->witnesses.empty() || nd->witnesses.front().root() != canonicalRoot)
nUnrecoverable++;
for (SaplingNoteData* nd : work)
nd->witnessRootValidated = true;
}
if (bulkRebuild) {
LogPrintf("%s: rebuilt %u note witness cache(s) to height %d in %ldms using %d thread(s)%s\n",
__func__, (unsigned)work.size(), tipHeight, (long)((tRead + tExtend) / 1000), nPar,
nUnrecoverable
? strprintf(" [WARNING: %d note(s) could not be rebuilt to the canonical anchor and were skipped]", nUnrecoverable).c_str()
nUnrecoverable.load()
? strprintf(" [WARNING: %d note(s) could not be rebuilt to the canonical anchor and were skipped]", nUnrecoverable.load()).c_str()
: "");
}
}
@@ -1913,6 +1981,8 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
bool fUpdated = false;
if (!fInsertedNew)
{
// Re-mined in a different block (a reorg): its notes sit at new tree positions.
bool fSameBlock = wtx.hashBlock.IsNull() || wtxIn.hashBlock.IsNull() || wtxIn.hashBlock == wtx.hashBlock;
// Merge
if (!wtxIn.hashBlock.IsNull() && wtxIn.hashBlock != wtx.hashBlock)
{
@@ -1925,7 +1995,7 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
wtx.nIndex = wtxIn.nIndex;
fUpdated = true;
}
if (UpdatedNoteData(wtxIn, wtx)) {
if (UpdatedNoteData(wtxIn, wtx, fSameBlock)) {
fUpdated = true;
}
if (wtxIn.fFromMe && wtxIn.fFromMe != wtx.fFromMe)
@@ -1962,29 +2032,92 @@ bool CWallet::AddToWallet(const CWalletTx& wtxIn, bool fFromLoadWallet, CWalletD
return true;
}
bool CWallet::UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx)
// True if both maps hold the same notes with identical persisted state (every field SaplingNoteData
// serializes). SaplingNoteData::operator== ignores the witnesses, so it cannot tell a real change
// from a redundant update.
// Copies the decrypted plaintext of each incoming note into the matching (same outpoint and ivk) entry that
// lacks one. UpdatedNoteData uses it when it keeps the wallet's entries: a replay or rescan has just
// decrypted the notes (FindMySaplingNotes), and entries loaded from wallet.dat have no plaintext. The cache
// is in memory only and outside every comparison, so this never makes a tx look changed.
static void KeepNotePlaintexts(const mapSaplingNoteData_t& incoming, mapSaplingNoteData_t& wallet)
{
for (const mapSaplingNoteData_t::value_type& in : incoming) {
if (!in.second.cachedPlaintext)
continue;
mapSaplingNoteData_t::iterator it = wallet.find(in.first);
if (it != wallet.end() && !it->second.cachedPlaintext && it->second.ivk == in.second.ivk)
it->second.cachedPlaintext = in.second.cachedPlaintext;
}
}
static bool SaplingNoteDataIdentical(const mapSaplingNoteData_t& a, const mapSaplingNoteData_t& b)
{
if (a.size() != b.size())
return false;
for (const mapSaplingNoteData_t::value_type& item : a) {
mapSaplingNoteData_t::const_iterator it = b.find(item.first);
if (it == b.end())
return false;
const SaplingNoteData& x = item.second;
const SaplingNoteData& y = it->second;
if (!(x.ivk == y.ivk) || x.nullifier != y.nullifier || x.witnessHeight != y.witnessHeight || x.witnesses != y.witnesses)
return false;
}
return true;
}
bool CWallet::UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock)
{
bool unchangedSaplingFlag = (wtxIn.mapSaplingNoteData.empty() || wtxIn.mapSaplingNoteData == wtx.mapSaplingNoteData);
if (!unchangedSaplingFlag) {
auto tmp = wtxIn.mapSaplingNoteData;
// Ensure we keep any cached witnesses we may already have
for (const std::pair <SaplingOutPoint, SaplingNoteData> nd : wtx.mapSaplingNoteData) {
if (tmp.count(nd.first) && nd.second.witnesses.size() > 0) {
tmp.at(nd.first).witnesses.assign(
nd.second.witnesses.cbegin(), nd.second.witnesses.cend());
//PART B2: only carry over witnessHeight TOGETHER with the witnesses it describes.
//Copying it unconditionally (when witnesses are NOT copied) advances the height past
//the actual witness state for a whole tx's notes at once = the batch desync originator.
tmp.at(nd.first).witnessHeight = nd.second.witnessHeight;
}
}
// Now copy over the updated note data
wtx.mapSaplingNoteData = tmp;
if (unchangedSaplingFlag) {
KeepNotePlaintexts(wtxIn.mapSaplingNoteData, wtx.mapSaplingNoteData);
return false;
}
return !unchangedSaplingFlag;
auto tmp = wtxIn.mapSaplingNoteData;
// Ensure we keep any cached witnesses we may already have
for (const mapSaplingNoteData_t::value_type& nd : wtx.mapSaplingNoteData) {
if (tmp.count(nd.first) && nd.second.witnesses.size() > 0) {
tmp.at(nd.first).witnesses.assign(
nd.second.witnesses.cbegin(), nd.second.witnesses.cend());
//PART B2: only carry over witnessHeight TOGETHER with the witnesses it describes.
//Copying it unconditionally (when witnesses are NOT copied) advances the height past
//the actual witness state for a whole tx's notes at once = the batch desync originator.
tmp.at(nd.first).witnessHeight = nd.second.witnessHeight;
//The nullifier is derived from the note and its witness position, so it travels with the
//witnesses too. wtxIn comes from FindMySaplingNotes, which never sets it; dropping it made
//VerifyAndSetInitialWitness clear and reseed the witnesses just carried over, so replaying
//or rescanning already-witnessed txs forced a rebuild of every one of their notes. A tx
//re-mined in another block is the exception: its notes moved, so the old nullifier is
//wrong and dropping it still reseeds them at the new position.
if (nd.second.nullifier && fSameBlock)
tmp.at(nd.first).nullifier = nd.second.nullifier;
}
}
// Keep a decrypted plaintext the wallet already holds (in memory only; a pure function of the
// outpoint and ivk) when the incoming note data lacks one.
for (const mapSaplingNoteData_t::value_type& nd : wtx.mapSaplingNoteData) {
mapSaplingNoteData_t::iterator it = tmp.find(nd.first);
if (it != tmp.end() && !it->second.cachedPlaintext && nd.second.cachedPlaintext && it->second.ivk == nd.second.ivk)
it->second.cachedPlaintext = nd.second.cachedPlaintext;
}
// A replayed or rescanned tx usually carries nothing new; don't rewrite it to disk. Do have
// VerifyAndSetInitialWitness re-check the kept witnesses against the active chain once (the
// flag is in memory only): the chain may have moved under them, and it keeps a rescan able to
// find and reseed a witness that went bad, without rebuilding the healthy ones.
if (SaplingNoteDataIdentical(tmp, wtx.mapSaplingNoteData)) {
for (mapSaplingNoteData_t::value_type& item : wtx.mapSaplingNoteData)
item.second.witnessRootValidated = false;
KeepNotePlaintexts(tmp, wtx.mapSaplingNoteData);
return false;
}
// Now copy over the updated note data
wtx.mapSaplingNoteData = tmp;
return true;
}
/**
@@ -2157,21 +2290,24 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
// Protocol Spec: 4.19 Block Chain Scanning (Sapling)
for (uint32_t i = 0; i < nZouts; ++i) {
const OutputDescription output = tx.vShieldedOutput[i];
const OutputDescription& output = tx.vShieldedOutput[i];
bool found = false;
for (auto it = mapSaplingFullViewingKeys.begin(); it != mapSaplingFullViewingKeys.end(); ++it) {
SaplingIncomingViewingKey ivk = it->first;
auto result = SaplingNotePlaintext::decrypt(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
auto result = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
if (result) {
auto address = ivk.address(result.get().d);
if (address && mapSaplingIncomingViewingKeys.count(address.get()) == 0) {
viewingKeysToAdd[address.get()] = ivk;
// The note's (d, pk_d) is the address it was sent to, exactly what ivk.address(d) would
// derive again: decrypt_with_note already computed pk_d from this ivk and d.
const libzcash::SaplingPaymentAddress address(result->second.d, result->second.pk_d);
if (mapSaplingIncomingViewingKeys.count(address) == 0) {
viewingKeysToAdd[address] = ivk;
}
// We don't cache the nullifier here as computing it requires knowledge of the note position
// in the commitment tree, which can only be determined when the transaction has been mined.
SaplingOutPoint op {hash, i};
SaplingNoteData nd;
nd.ivk = ivk;
nd.cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
noteData.insert(std::make_pair(op, nd));
found = true;
break;
@@ -2180,7 +2316,7 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
if (!found) {
for (auto it = mapSaplingIncomingViewingKeys.begin(); it != mapSaplingIncomingViewingKeys.end(); ++it) {
SaplingIncomingViewingKey ivk = it-> second;
auto result = SaplingNotePlaintext::decrypt(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
auto result = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
if (!result) {
continue;
}
@@ -2189,6 +2325,7 @@ std::pair<mapSaplingNoteData_t, SaplingIncomingViewingKeyMap> CWallet::FindMySap
SaplingOutPoint op {hash, i};
SaplingNoteData nd;
nd.ivk = ivk;
nd.cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
noteData.insert(std::make_pair(op, nd));
break;
}
@@ -5609,6 +5746,137 @@ bool CMerkleTx::AcceptToMemoryPool(bool fLimitFree, bool fRejectAbsurdFee)
return ::AcceptToMemoryPool(mempool, state, *this, fLimitFree, NULL, fRejectAbsurdFee);
}
unsigned int CWallet::FillSaplingNoteCache(const std::vector<SaplingNoteToDecrypt>& notes)
{
AssertLockHeld(cs_wallet);
// Each decryption is independent: it reads its own output and writes only its own note's
// cachedPlaintext. So a big batch (a whole wallet after a restart, ~0.5ms a note) is split across
// threads the way BuildWitnessCache's Phase 2 is, while this thread keeps cs_wallet held. Workers
// must not lock, or even assert a lock: under DEBUG_LOCKORDER a fresh thread has no lock stack.
// Spawning costs ~1ms, so small batches stay serial.
int nPar = 1;
if (notes.size() >= 128) {
nPar = (int)GetArg("-witnessbuildthreads", 0);
if (nPar <= 0) nPar = (int)std::thread::hardware_concurrency();
nPar = std::max(1, std::min(nPar, (int)(notes.size() / 64)));
}
std::vector<unsigned int> filled(nPar, 0);
auto worker = [&](int tid) {
for (size_t k = (size_t)tid; k < notes.size(); k += (size_t)nPar) {
const SaplingNoteToDecrypt& n = notes[k];
try {
auto result = SaplingNotePlaintext::decrypt_with_note(
n.output->encCiphertext, n.nd->ivk, n.output->ephemeralKey, n.output->cm);
if (result) {
n.nd->cachedPlaintext = SaplingNoteCache{ result->second, result->first.memo() };
filled[tid]++;
}
} catch (...) {
// Left unfilled, exactly like a note that doesn't decrypt; callers handle that. Nothing may
// escape a std::thread.
}
}
};
if (nPar <= 1) {
worker(0);
} else {
// If the OS refuses a thread, run the remaining shards here instead of letting the exception
// unwind past joinable threads (std::terminate). Each shard still runs exactly once.
std::vector<std::thread> threads;
threads.reserve(nPar);
int t = 0;
try {
for (; t < nPar; t++) threads.emplace_back(worker, t);
} catch (const std::exception& e) {
LogPrintf("%s: could only start %d of %d threads (%s); decrypting the rest inline\n", __func__, t, nPar, e.what());
}
for (int u = t; u < nPar; u++) worker(u);
for (std::thread& th : threads) th.join();
}
unsigned int nFilled = 0;
for (unsigned int f : filled)
nFilled += f;
nSaplingNoteCacheFills += nFilled;
nSaplingNoteCacheFillThreads = nPar;
LogPrint("bench", "%s: decrypted %u of %u notes using %d thread(s)\n", __func__, nFilled, (unsigned int)notes.size(), nPar);
return nFilled;
}
// wallet.dat doesn't store the plaintext, so without this the first balance call after a restart decrypts
// the whole wallet under cs_main (~0.5ms a note). init runs it before RPC leaves warmup and before the
// wallet notification thread starts; only the few RPCs allowed during warmup (key export, address
// listing) can wait on cs_wallet meanwhile. Unlike GetFilteredNotes it never asserts, so a corrupt note
// can't stop the node from starting (or from serving those RPCs); GetFilteredNotes still asserts on such
// a note, as before.
CWallet::SaplingNoteCacheWarmup CWallet::WarmSaplingNoteCache()
{
const std::chrono::steady_clock::time_point start = std::chrono::steady_clock::now();
LOCK(cs_wallet);
SaplingNoteCacheWarmup result;
std::vector<SaplingNoteToDecrypt> uncached;
for (auto& p : mapWallet) {
CWalletTx& wtx = p.second;
for (auto& item : wtx.mapSaplingNoteData) {
result.notes++;
if (item.second.cachedPlaintext)
continue;
if (item.first.n >= wtx.vShieldedOutput.size()) {
LogPrint("zrpcunsafe", "%s: note %s:%u has no such output\n", __func__, item.first.hash.ToString(), item.first.n);
result.noSuchOutput++;
continue;
}
uncached.push_back(SaplingNoteToDecrypt { &item.second, &wtx.vShieldedOutput[item.first.n] });
}
}
result.filled = uncached.empty() ? 0 : FillSaplingNoteCache(uncached);
for (const SaplingNoteToDecrypt& n : uncached) {
if (!n.nd->cachedPlaintext) {
LogPrint("zrpcunsafe", "%s: a note of output %s does not decrypt under its ivk\n", __func__, n.output->cm.ToString());
result.undecryptable++;
}
}
const unsigned int nBad = result.noSuchOutput + result.undecryptable;
const int64_t nMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - start).count();
LogPrintf("%s: decrypted %u of %u Sapling notes in %dms%s\n", __func__, result.filled, result.notes, nMs,
nBad ? strprintf("; %u could not be decrypted (-debug=zrpcunsafe lists them)", nBad) : "");
return result;
}
// -checknotecache: decrypts a cached note again, and logs and repairs any difference. GetFilteredNotes runs
// it on every note that passes its tx, spent and locked filters, before the address filter, so a corrupt
// cached address can't wrongly drop a note unchecked. A diagnostic for the cache: it costs the full
// decryption the cache exists to avoid.
static void CheckSaplingNoteCache(const SaplingOutPoint& op, SaplingNoteData& nd, const OutputDescription& output)
{
auto fresh = SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, nd.ivk, output.ephemeralKey, output.cm);
if (!fresh) {
LogPrintf("%s: note %s:%u has a cached plaintext but no longer decrypts\n", __func__, op.hash.ToString(), op.n);
return;
}
const SaplingNoteCache& cached = *nd.cachedPlaintext;
const libzcash::SaplingNote& note = fresh->second;
if (!(cached.note.d == note.d && cached.note.pk_d == note.pk_d && cached.note.r == note.r &&
cached.note.value() == note.value() && cached.memo == fresh->first.memo())) {
LogPrintf("%s: cached plaintext of note %s:%u differs from a fresh decryption; using the fresh one\n",
__func__, op.hash.ToString(), op.n);
nd.cachedPlaintext = SaplingNoteCache{ note, fresh->first.memo() };
}
}
// Replaces tx_height() on the wallet's note paths. tx_height() re-read the tx from disk through
// GetTransaction, once per wallet tx (or per returned note) under LOCK2, and on a -txindex=0 node it
// never found a fully shielded tx, so those notes were silently skipped. The depth already locates
// the block: GetDepthInMainChain is chainActive.Height() - height + 1 for an active-chain tx.
int HeightFromDepth(int nDepth)
{
AssertLockHeld(cs_main);
return nDepth > 0 ? chainActive.Height() - nDepth + 1 : 0;
}
/**
* Find notes in the wallet filtered by payment address, min depth and ability to spend.
* These notes are decrypted and added to the output parameter vector, outEntries.
@@ -5645,79 +5913,126 @@ void CWallet::GetFilteredNotes(
{
LOCK2(cs_main, cs_wallet);
LogPrintf("%s ignoreLocked=%d\n", __func__, ignoreLocked);
// Every caller already holds LOCK2, so the lock above never waits and this times the walk alone.
// Time spent waiting for the locks shows up in the RPC's own "took" line (rpc/server.cpp).
const std::chrono::steady_clock::time_point walkStart = std::chrono::steady_clock::now();
const size_t nEntriesBefore = saplingEntries.size();
unsigned int nTxs = 0, nNotes = 0, nSpent = 0;
// Pass 1: the notes that pass every filter not needing the plaintext, in wallet order.
struct Candidate {
const SaplingOutPoint* op;
SaplingNoteData* nd;
const OutputDescription* output;
int nDepth;
};
std::vector<Candidate> candidates;
std::vector<SaplingNoteToDecrypt> uncached;
for (auto & p : mapWallet) {
CWalletTx wtx = p.second;
// By reference: a copy deep-copies every note's witness list (up to WITNESS_CACHE_SIZE
// witnesses each), and it was made before any filter ran. Not const: a missing plaintext is
// cached onto the wallet's own entry.
CWalletTx& wtx = p.second;
// Most wallet txs (coinbase, sends from other wallets) carry no notes of ours.
if (wtx.mapSaplingNoteData.empty())
continue;
// Filter the transactions before checking for notes
if (!CheckFinalTx(wtx) || wtx.GetBlocksToMaturity() > 0)
continue;
// cs_main is held, so the depth can't change during the walk.
const int nDepth = wtx.GetDepthInMainChain();
if (minDepth > 1) {
int nHeight = tx_height(wtx.GetHash());
int nHeight = HeightFromDepth(nDepth);
if ( nHeight == 0 ) {
continue;
}
int nDepth = wtx.GetDepthInMainChain();
int dpowconfs = hush_dpowconfs(nHeight,nDepth);
if ( dpowconfs < minDepth || dpowconfs > maxDepth) {
continue;
}
} else {
if ( wtx.GetDepthInMainChain() < minDepth ||
wtx.GetDepthInMainChain() > maxDepth) {
if ( nDepth < minDepth || nDepth > maxDepth) {
continue;
}
}
nTxs++;
for (auto & pair : wtx.mapSaplingNoteData) {
SaplingOutPoint op = pair.first;
SaplingNoteData nd = pair.second;
auto maybe_pt = SaplingNotePlaintext::decrypt(
wtx.vShieldedOutput[op.n].encCiphertext,
nd.ivk,
wtx.vShieldedOutput[op.n].ephemeralKey,
wtx.vShieldedOutput[op.n].cm);
assert(static_cast<bool>(maybe_pt));
auto notePt = maybe_pt.get();
auto maybe_pa = nd.ivk.address(notePt.d);
assert(static_cast<bool>(maybe_pa));
auto pa = maybe_pa.get();
// skip notes which belong to a different payment address in the wallet
if (!(filterAddresses.empty() || filterAddresses.count(pa))) {
continue;
}
const SaplingOutPoint& op = pair.first;
SaplingNoteData& nd = pair.second;
nNotes++;
// The cheap filters run before any decryption, which costs ~0.5ms of Jubjub arithmetic
// per note. Spent notes stay in the wallet (-deletetx is off by default).
if (ignoreSpent && nd.nullifier && IsSaplingSpent(*nd.nullifier)) {
nSpent++;
continue;
}
// skip notes which cannot be spent
if (requireSpendingKey) {
libzcash::SaplingIncomingViewingKey ivk;
libzcash::SaplingFullViewingKey fvk;
if (!(GetSaplingIncomingViewingKey(pa, ivk) &&
GetSaplingFullViewingKey(ivk, fvk) &&
HaveSaplingSpendingKey(fvk))) {
continue;
}
}
// skip locked notes
if (ignoreLocked && IsLockedNote(op)) {
LogPrintf("%s: skipping locked note %s\n", __func__, op.hash.ToString().substr(0,10).c_str());
continue;
}
if (ignoreLocked && IsLockedNote(op)) {
LogPrint("zrpcunsafe", "%s: skipping locked note %s\n", __func__, op.hash.ToString().substr(0,10).c_str());
continue;
}
auto note = notePt.note(nd.ivk).get();
saplingEntries.push_back(SaplingNoteEntry {
op, pa, note, notePt.memo(), wtx.GetDepthInMainChain() });
assert(op.n < wtx.vShieldedOutput.size());
const OutputDescription* output = &wtx.vShieldedOutput[op.n];
candidates.push_back(Candidate { &op, &nd, output, nDepth });
if (!nd.cachedPlaintext)
uncached.push_back(SaplingNoteToDecrypt { &nd, output });
}
}
// Decrypt what the cache lacks: normally nothing, since FindMySaplingNotes fills new notes and the
// startup warm-up fills the ones loaded from wallet.dat.
const unsigned int nFilled = uncached.empty() ? 0 : FillSaplingNoteCache(uncached);
// Pass 2: the filters that need the note's address.
if (filterAddresses.empty())
saplingEntries.reserve(saplingEntries.size() + candidates.size());
for (const Candidate& c : candidates) {
// A wallet note that doesn't decrypt under its own ivk is corrupt; as before, that's fatal here.
assert(static_cast<bool>(c.nd->cachedPlaintext));
if (fCheckNoteCache)
CheckSaplingNoteCache(*c.op, *c.nd, *c.output);
const SaplingNoteCache& plaintext = *c.nd->cachedPlaintext;
const libzcash::SaplingPaymentAddress pa(plaintext.note.d, plaintext.note.pk_d);
// skip notes which belong to a different payment address in the wallet
if (!(filterAddresses.empty() || filterAddresses.count(pa))) {
continue;
}
// skip notes which cannot be spent
if (requireSpendingKey) {
libzcash::SaplingIncomingViewingKey ivk;
libzcash::SaplingFullViewingKey fvk;
if (!(GetSaplingIncomingViewingKey(pa, ivk) &&
GetSaplingFullViewingKey(ivk, fvk) &&
HaveSaplingSpendingKey(fvk))) {
continue;
}
}
saplingEntries.push_back(SaplingNoteEntry {
*c.op, pa, plaintext.note, plaintext.memo, c.nDepth });
}
// A slow walk holds cs_main long enough to stall block connection, so it is always logged; a
// fast one only with -debug=bench (the balance RPCs run it on every GUI poll). The wallet-size
// counts go to the default log only with -debug=bench too.
const int64_t nWalkMs = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - walkStart).count();
if (LogAcceptCategory("bench")) {
LogPrintf("%s: minDepth=%d ignoreSpent=%d requireSpendingKey=%d ignoreLocked=%d txs=%u/%u notes=%u spent=%u decrypted=%u entries=%u walk=%dms\n",
__func__, minDepth, ignoreSpent, requireSpendingKey, ignoreLocked, nTxs, (unsigned int)mapWallet.size(),
nNotes, nSpent, nFilled, (unsigned int)(saplingEntries.size() - nEntriesBefore), nWalkMs);
} else if (nWalkMs >= 500) {
LogPrintf("%s: minDepth=%d ignoreSpent=%d requireSpendingKey=%d ignoreLocked=%d walk=%dms\n",
__func__, minDepth, ignoreSpent, requireSpendingKey, ignoreLocked, nWalkMs);
}
}
src/wallet/wallet.h
+55 -2
View File
@@ -62,6 +62,8 @@ extern bool bSpendZeroConfChange;
extern bool fSendFreeTransactions;
extern bool fPayAtLeastCustomFee;
extern bool fTxDeleteEnabled;
//! -checknotecache: re-decrypt cached Sapling notes in GetFilteredNotes and repair any difference
extern bool fCheckNoteCache;
extern bool fTxConflictDeleteEnabled;
extern int fDeleteInterval;
extern unsigned int fDeleteTransactionsAfterNBlocks;
@@ -301,6 +303,14 @@ class SproutNoteData
};
/** A Sapling note's decrypted plaintext as the wallet uses it: the note (value, rcm, and the address it
* was sent to as (d, pk_d)) and its memo. */
struct SaplingNoteCache
{
libzcash::SaplingNote note;
std::array<unsigned char, HUSH_MEMO_SIZE> memo;
};
class SaplingNoteData
{
public:
@@ -328,6 +338,15 @@ public:
// read true and short-circuit the witness self-heal in VerifyAndSetInitialWitness.
bool witnessRootValidated = false;
//In Memory Only
// The note's decrypted plaintext, so the balance and note-listing calls don't redo the Sapling
// decryption (several Jubjub scalar multiplications) for every note on every call. It is a pure
// function of (txid, output index, ivk), so nothing invalidates it. Never serialized, and left out
// of operator== and SaplingNoteDataIdentical on purpose: a filled and an unfilled entry persist the
// same state, so filling one must never make a tx look changed and get rewritten. Filled by
// FindMySaplingNotes when the note is found, otherwise by FillSaplingNoteCache.
boost::optional<SaplingNoteCache> cachedPlaintext;
ADD_SERIALIZE_METHODS;
template <typename Stream, typename Operation>
@@ -366,6 +385,11 @@ struct SaplingNoteEntry
int confirmations;
};
/** Height of the active-chain block holding a wallet tx that is nDepth deep (as GetDepthInMainChain
* returns it, and SaplingNoteEntry::confirmations holds it), or 0 when the tx is unconfirmed or not
* in the active chain. Caller holds cs_main. */
int HeightFromDepth(int nDepth);
/** A transaction with a merkle branch linking it to the block chain. */
class CMerkleTx : public CTransaction
{
@@ -895,7 +919,7 @@ public:
protected:
int SaplingWitnessMinimumHeight(const uint256& nullifier, int nWitnessHeight, int nMinimumHeight);
int SaplingWitnessMinimumHeight(const boost::optional<uint256>& nullifier, int nWitnessHeight, int nMinimumHeight);
/**
* pindex is the new tip being connected.
@@ -965,8 +989,12 @@ private:
template <class T>
void SyncMetaData(std::pair<typename TxSpendMap<T>::iterator, typename TxSpendMap<T>::iterator>);
// Last block ThreadNotifyWallets delivered to ChainTip, i.e. the block the wallet's state
// reflects (guarded by cs_wallet). SetBestChain stamps this instead of the chain tip.
const CBlockIndex* pindexWalletTip = nullptr;
protected:
bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx);
bool UpdatedNoteData(const CWalletTx& wtxIn, CWalletTx& wtx, bool fSameBlock);
void MarkAffectedTransactionsDirty(const CTransaction& tx);
/* the hd chain data model (chain counters) */
@@ -1294,6 +1322,8 @@ public:
bool CommitAutomatedTx(const CTransaction& tx);
/** Saves witness caches and best block locator to disk. Overrides CValidationInterface. */
void SetBestChain(const CBlockLocator& loc);
// Seed the block the wallet's state reflects, before ThreadNotifyWallets starts delivering.
void SetWalletTip(const CBlockIndex* pindex);
/** As SetBestChain, but for use INSIDE a long scan: opens the wallet DB with fFlushOnClose=false
* so the call does not trigger a full BDB txn_checkpoint over the whole cache (see the comment
* at CWallet::SetBestChain), and reports whether the write actually committed. */
@@ -1458,6 +1488,29 @@ public:
bool LoadCryptedMnemonicEntropy(const uint256& entropyFp, const std::vector<unsigned char>& vchCryptedSecret);
/* Find notes filtered by payment address, min depth, ability to spend */
/** A wallet note whose decrypted plaintext is not cached yet, with the output it decrypts. */
struct SaplingNoteToDecrypt {
SaplingNoteData* nd;
const OutputDescription* output;
};
/** Decrypts each note into nd->cachedPlaintext and returns how many it filled. A note that doesn't
* decrypt under its ivk is left unfilled. Caller holds cs_wallet. */
unsigned int FillSaplingNoteCache(const std::vector<SaplingNoteToDecrypt>& notes);
/** Notes FillSaplingNoteCache has decrypted since startup (FindMySaplingNotes' fills not counted), and
* the threads its last call used. Guarded by cs_wallet. */
uint64_t nSaplingNoteCacheFills = 0;
int nSaplingNoteCacheFillThreads = 0;
struct SaplingNoteCacheWarmup {
unsigned int notes = 0; //!< notes in the wallet
unsigned int filled = 0; //!< decrypted by this call
unsigned int noSuchOutput = 0; //!< naming an output their tx doesn't have
unsigned int undecryptable = 0; //!< not decrypting under their own ivk
};
/** Decrypts every wallet note whose plaintext isn't cached yet (all of them after wallet.dat is loaded).
* Never asserts: a note that can't be decrypted is counted, logged and left unfilled. Run at startup. */
SaplingNoteCacheWarmup WarmSaplingNoteCache();
void GetFilteredNotes(std::vector<SaplingNoteEntry>& saplingEntries,
std::string address,
int minDepth=1,
src/zcash/Note.cpp
+37 -3
View File
@@ -115,11 +115,16 @@ boost::optional<SaplingOutgoingPlaintext> SaplingOutgoingPlaintext::decrypt(
}
}
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
namespace {
// SaplingNotePlaintext::decrypt with an incoming viewing key, also handing back the pk_d = [ivk]g_d
// it derives to check cmu.
boost::optional<SaplingNotePlaintext> DecryptWithIvk(
const SaplingEncCiphertext &ciphertext,
const uint256 &ivk,
const uint256 &epk,
const uint256 &cmu
const uint256 &cmu,
uint256 &pk_d
)
{
auto pt = AttemptSaplingEncDecryption(ciphertext, ivk, epk);
@@ -140,7 +145,6 @@ boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
return boost::none;
}
uint256 pk_d;
if (!librustzcash_ivk_to_pkd(ivk.begin(), ret.d.data(), pk_d.begin())) {
return boost::none;
}
@@ -164,6 +168,36 @@ boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
return ret;
}
}
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
const SaplingEncCiphertext &ciphertext,
const uint256 &ivk,
const uint256 &epk,
const uint256 &cmu
)
{
uint256 pk_d;
return DecryptWithIvk(ciphertext, ivk, epk, cmu, pk_d);
}
boost::optional<std::pair<SaplingNotePlaintext, SaplingNote>> SaplingNotePlaintext::decrypt_with_note(
const SaplingEncCiphertext &ciphertext,
const uint256 &ivk,
const uint256 &epk,
const uint256 &cmu
)
{
uint256 pk_d;
auto pt = DecryptWithIvk(ciphertext, ivk, epk, cmu, pk_d);
if (!pt) {
return boost::none;
}
// ivk_to_pkd succeeding implies d is a valid diversifier, which is all note(ivk) checks on top.
SaplingNote note(pt->d, pk_d, pt->value(), pt->rcm);
return std::make_pair(*pt, note);
}
boost::optional<SaplingNotePlaintext> SaplingNotePlaintext::decrypt(
const SaplingEncCiphertext &ciphertext,
const uint256 &epk,
src/zcash/Note.hpp
+11
View File
@@ -85,6 +85,17 @@ public:
const uint256 &cmu
);
// decrypt(ciphertext, ivk, epk, cmu) that also returns the note. Building the note needs
// pk_d = [ivk]g_d, which the decryption derives anyway to check cmu; note(ivk) would derive it
// again (another Jubjub scalar multiplication). A distinct name rather than an overload, so it
// can never capture a call meant for the (epk, esk, pk_d, cmu) one.
static boost::optional<std::pair<SaplingNotePlaintext, SaplingNote>> decrypt_with_note(
const SaplingEncCiphertext &ciphertext,
const uint256 &ivk,
const uint256 &epk,
const uint256 &cmu
);
boost::optional<SaplingNote> note(const SaplingIncomingViewingKey& ivk) const;
virtual ~SaplingNotePlaintext() {}