The block scan appended each shielded output's leaf to the witness of
every live (unspent, not pending) note as it went, one witness at a time
on the sync thread, while holding the wallet's txs write lock: live
notes x outputs appends per block, all on one core.
With 16 or more live notes, the scan now collects the block's leaves in
output order and appends them to those witnesses once, at the end of the
block, still under the same lock and before anything reads them. When
that is at least 128 appends, the witnesses are split into disjoint
chunks over min(8, cores, live notes / 8) scoped threads; otherwise the
appends run inline. Every witness receives the same leaves in the same
order, including a leaf whose epk does not parse; an output whose cmu
does not parse still adds nothing. The hashing work is the same, only
spread over cores; the speed-up has not been benchmarked yet.
No output change. Each qualifying block starts up to min(8, cores)
short-lived threads.
Tests check that the threaded helper matches the plain double loop byte
for byte for 0-64 witnesses, and that the deferred path handles bad-epk
and bad-cmu outputs. The 1000-seed oracle passes, as do the suite and
125 oracle seeds pinned to 2 cores, and a differential run against the
unmodified backend is byte-identical, apart from the lists cleared two
commits earlier, for wallets below and above 16 live notes and blocks of
0, 1 and 50 outputs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The block scan replaced each note's witness list with the result of
split_off(), which allocates a new buffer on every call: every block,
each note with history paid an allocation and, once its list was full, a
copy of all 100 entries.
The duplicate of the newest witness is now pushed and, when the list
exceeds 100 entries, the oldest is drained in place; notes with no
history are skipped. The list keeps the same entries in the same order,
so nothing in the output changes.
The witness tests and the oracle pass, and a differential run against
the unmodified backend is byte-identical (every witness entry and the
whole saved file after every operation) apart from the lists the
previous commit clears.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every note keeps a list of up to 100 witnesses, one per block, so that a
reorg can roll it back; a full list takes roughly 40-90 KB of the wallet
file. The block scan kept duplicating and trimming that list for every
note that had one, spent notes included, although a spent note never
reads its history again once its spend can no longer be rolled back. So
every note the wallet ever spent kept costing memory, per-block scan
time and file size.
Once the block window is full (101 blocks), a spent note whose spending
transaction's block is below the window's first block now gets an empty
witness list. This runs before each block's witness update and once when
a wallet is loaded. Such a spend can only be undone by a reorg that pops
the whole window, and that already empties every witness list, so reorgs
behave exactly as before.
Behaviour changes:
- notes whose spend lies below a full reorg window lose their witness
history in memory, at load and in the next save; existing large
wallets shrink on their first save (which, like every save, copies the
previous file to wallet.dat.bak);
- the wallet format (v9) is unchanged; old builds and SilentDragonXLite
read an empty list as an unspendable note, which a spent note already
is;
- new info log at load: "Dropped witness history of N notes whose spend
is deeper than 100 blocks".
The oracle now also accepts an empty list for a spent note that was
cleared before a later partial reorg.
New witness tests compare every block's lists with a model of the old
algorithm, including reorgs across the clear point and an old file with
full lists, which loads and re-saves to the new bytes. The 1000-seed
oracle run passes (12,207 reorgs, 993 sends verified offline), and a
differential run against the unmodified backend is byte-identical apart
from exactly the cleared lists; the old build reads the new files, and
17 sends it built from them verified offline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
note_address() re-derived a note's payment address from its viewing key
and diversifier (a group hash, an ivk hash and a 252-bit scalar
multiplication) for every note in `notes` and `list` and for each new
note in the block scan. The z-balance sums did the same derivation in
their address filter, so `balance`, which sums four balances for each of
Z z-addresses, cost Z x (N + 2U) derivations for N notes, U of them
unspent. do_list_notes also encoded up to Z + Z^2 addresses to find the
spendable ones and cloned that set once per transaction.
Now:
- note_address() encodes the note's stored pk_d with its diversifier.
Both decryption and wallet load build that pk_d from the same key and
diversifier, so the string is byte-identical;
- the three z-balance sums share one address filter. The per-address
sums behind `balance` compare (diversifier, pk_d) with each z-address
directly; any other address string keeps the old exact string compare;
- those sums work from a copy of the z-addresses whose lock is released
before any note is read, so the txs-then-zkeys lock order that sync
uses is unchanged;
- do_list_notes builds the spendable-address set with Z encodes and
borrows it.
No output changes: balances and the notes/list/balance JSON are
byte-identical. The address filter no longer contains an unwrap (it
could not fail).
Tests compare against verbatim copies of the old functions over
randomized chains (note addresses, also after a reload; every filtered
balance for own, diversified, foreign and junk address strings; the
per-address balances; the spendable set), and a concurrency test fails
with a deadlock if the zkeys lock is held across the sums. A
differential run against the unmodified backend compared per-address
balances and the JSON after every operation, through reorgs and reloads,
and the old build re-saves the new files byte for byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No existing test checks that a send's anchor, nullifiers and proofs are
valid, or that note witnesses stay right through reorgs and reloads, and
that is what the following witness and note-selection changes touch.
oracle_tests.rs drives a LightWallet through its public entry points
over seeded random fake chains that start at real mainnet checkpoints:
receives to HD, diversified, imported, view-only and foreign keys,
zero-value and dust notes, outputs with an unparseable epk, compact
spends with change, real sends that are mined later or never (mempool
expiry), reorgs of 1-100 blocks in one call or block by block (re-mining
spends on the new branch), and save/reload.
An independent reference keeps its own commitment tree, every block's
root and every note's true witness at each height. After each step it
checks the tree and block window, every note's fields and spent state,
witness-list lengths, every retained witness of a live note byte for
byte (sampled ones also by root, position and auth path), the balances
and the mempool. Sends are captured in the broadcast closure, never
sent, and verified offline: the anchor must be the reference root at tip
minus the anchor offset, every nullifier must belong to an eligible
note, and the Sapling verifier must accept every spend and output proof
and the binding signature (the first send of a run also checks that it
rejects a wrong anchor and a wrong nullifier).
The default run (20 seeds x 300 blocks) is part of the normal suite; a
1000-seed x 400-block run and a 20-seed x 5000-block soak are #[ignore].
Every run prints its seed, and a single seed can be re-run.
Two pre-existing bugs are documented as #[ignore] reproducers and kept
out of the green runs:
- a reorg while a send is pending panics the backend (invalidate_block
unwraps `spent` on a note that only has `unconfirmed_spent`);
- a pending send's notes stop receiving witness updates, so after a
restart (the send never mined) or a reorg that undoes the block that
mined it, they are spendable again with stale witnesses, and later
sends from them fail until a rescan.
On the unchanged production code the default and full runs pass (1000
seeds: 12,194 reorgs up to 100 deep, 5,893 reloads, 993 proving sends
verified offline at anchor offsets 0 and 4) and the reproducers fail as
described.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Vendor the two local Rust crates that build the lite backend artifact into
third_party/silentdragonxlite/ (the qtlib C-ABI wrapper + the silentdragonxlitelib
core, with proto/res and all the lite-send fixes), and point
build-lite-backend-artifact.sh's default --backend-dir there, so the lite wallet
builds without the upstream SilentDragonXLite repo.
External build inputs are now only the Rust toolchain + git.dragonx.is + crates.io:
- the 6 librustzcash git deps point at the git.dragonx.is/DragonX/librustzcash
mirror (pinned rev acff1444), not git.hush.is;
- the Sapling params are gitignored (not committed, no Git LFS) - the build fetches
them from the git.dragonx.is/DragonX/zcash-params 'sapling-v1' release and verifies
their SHA-256 before rust-embed bakes them in (ensure_sapling_params).
For fully offline builds, cargo vendor into lib/vendor/ and add a vendored-sources
redirect (vendor/ is gitignored; the script symlinks it into the prepared dir).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DanSandClaude Opus 5.5