Every note keeps a list of up to 100 witnesses, one per block, so that a
reorg can roll it back; a full list takes roughly 40-90 KB of the wallet
file. The block scan kept duplicating and trimming that list for every
note that had one, spent notes included, although a spent note never
reads its history again once its spend can no longer be rolled back. So
every note the wallet ever spent kept costing memory, per-block scan
time and file size.
Once the block window is full (101 blocks), a spent note whose spending
transaction's block is below the window's first block now gets an empty
witness list. This runs before each block's witness update and once when
a wallet is loaded. Such a spend can only be undone by a reorg that pops
the whole window, and that already empties every witness list, so reorgs
behave exactly as before.
Behaviour changes:
- notes whose spend lies below a full reorg window lose their witness
history in memory, at load and in the next save; existing large
wallets shrink on their first save (which, like every save, copies the
previous file to wallet.dat.bak);
- the wallet format (v9) is unchanged; old builds and SilentDragonXLite
read an empty list as an unspendable note, which a spent note already
is;
- new info log at load: "Dropped witness history of N notes whose spend
is deeper than 100 blocks".
The oracle now also accepts an empty list for a spent note that was
cleared before a later partial reorg.
New witness tests compare every block's lists with a model of the old
algorithm, including reorgs across the clear point and an old file with
full lists, which loads and re-saves to the new bytes. The 1000-seed
oracle run passes (12,207 reorgs, 993 sends verified offline), and a
differential run against the unmodified backend is byte-identical apart
from exactly the cleared lists; the old build reads the new files, and
17 sends it built from them verified offline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every LightClient copied the Sapling spend and output parameters, which
rust-embed already compiles into the binary, onto the heap: 51,551,256
bytes (49.2 MiB) of private memory per live client, whether or not it
ever sent anything.
LightClient::sapling_output and sapling_spend are now Cow<'static, [u8]>
and borrow the embedded image. set_sapling_params still stores an owned
copy, with its is-empty guard, hash checks and disk write unchanged. The
prover receives the same bytes.
Behaviour changes:
- about 49 MiB less anonymous memory per live LightClient; the wallet
file and JSON are unchanged;
- the first send or shield pages the 51.5 MB of embedded params in from
the binary, where client creation used to copy them (an estimated
0.3-1 s from the AppImage);
- the public fields sapling_output and sapling_spend change type from
Vec<u8> to Cow<'static, [u8]>: a Rust API change, with no C ABI or
exported-symbol change.
tests/sapling_params.rs checks that a restored, a new and a reloaded
client all borrow the embedded image (same pointer, pinned lengths and
SHA-256), that set_sapling_params keeps its guard and hash checks, and
that the borrowed params parse like a heap copy. An ignored test
measures 576 KiB of RssAnon growth for a second client, against 46.5 MiB
with the old heap copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
note_address() re-derived a note's payment address from its viewing key
and diversifier (a group hash, an ivk hash and a 252-bit scalar
multiplication) for every note in `notes` and `list` and for each new
note in the block scan. The z-balance sums did the same derivation in
their address filter, so `balance`, which sums four balances for each of
Z z-addresses, cost Z x (N + 2U) derivations for N notes, U of them
unspent. do_list_notes also encoded up to Z + Z^2 addresses to find the
spendable ones and cloned that set once per transaction.
Now:
- note_address() encodes the note's stored pk_d with its diversifier.
Both decryption and wallet load build that pk_d from the same key and
diversifier, so the string is byte-identical;
- the three z-balance sums share one address filter. The per-address
sums behind `balance` compare (diversifier, pk_d) with each z-address
directly; any other address string keeps the old exact string compare;
- those sums work from a copy of the z-addresses whose lock is released
before any note is read, so the txs-then-zkeys lock order that sync
uses is unchanged;
- do_list_notes builds the spendable-address set with Z encodes and
borrows it.
No output changes: balances and the notes/list/balance JSON are
byte-identical. The address filter no longer contains an unwrap (it
could not fail).
Tests compare against verbatim copies of the old functions over
randomized chains (note addresses, also after a reload; every filtered
balance for own, diversified, foreign and junk address strings; the
per-address balances; the spendable set), and a concurrency test fails
with a deadlock if the zkeys lock is held across the sums. A
differential run against the unmodified backend compared per-address
balances and the JSON after every operation, through reorgs and reloads,
and the old build re-saves the new files byte for byte.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No existing test checks that a send's anchor, nullifiers and proofs are
valid, or that note witnesses stay right through reorgs and reloads, and
that is what the following witness and note-selection changes touch.
oracle_tests.rs drives a LightWallet through its public entry points
over seeded random fake chains that start at real mainnet checkpoints:
receives to HD, diversified, imported, view-only and foreign keys,
zero-value and dust notes, outputs with an unparseable epk, compact
spends with change, real sends that are mined later or never (mempool
expiry), reorgs of 1-100 blocks in one call or block by block (re-mining
spends on the new branch), and save/reload.
An independent reference keeps its own commitment tree, every block's
root and every note's true witness at each height. After each step it
checks the tree and block window, every note's fields and spent state,
witness-list lengths, every retained witness of a live note byte for
byte (sampled ones also by root, position and auth path), the balances
and the mempool. Sends are captured in the broadcast closure, never
sent, and verified offline: the anchor must be the reference root at tip
minus the anchor offset, every nullifier must belong to an eligible
note, and the Sapling verifier must accept every spend and output proof
and the binding signature (the first send of a run also checks that it
rejects a wrong anchor and a wrong nullifier).
The default run (20 seeds x 300 blocks) is part of the normal suite; a
1000-seed x 400-block run and a 20-seed x 5000-block soak are #[ignore].
Every run prints its seed, and a single seed can be re-run.
Two pre-existing bugs are documented as #[ignore] reproducers and kept
out of the green runs:
- a reorg while a send is pending panics the backend (invalidate_block
unwraps `spent` on a note that only has `unconfirmed_spent`);
- a pending send's notes stop receiving witness updates, so after a
restart (the send never mined) or a reorg that undoes the block that
mined it, they are spendable again with stale witnesses, and later
sends from them fail until a rescan.
On the unchanged production code the default and full runs pass (1000
seeds: 12,194 reorgs up to 100 deep, 5,893 reloads, 993 proving sends
verified offline at anchor offsets 0 and 4) and the reproducers fail as
described.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lite backend core's in-crate tests no longer compiled (119 errors in
lightwallet/tests.rs and the lightclient and walletzkey test modules):
they still used the old LightWallet::new and send_to_address signatures,
the removed extfvks/extsks/zaddress fields and the old new_from_phrase
and do_send arities. So `cargo test` built nothing and no backend change
could be checked against them.
Port them, and the commands and checkpoints tests, to the current API,
touching only #[cfg(test)] code:
- LightWallet::new takes 4 arguments, send_to_address 7 (with an
explicit default fee, returning (txid, raw tx)), new_from_phrase 5 and
do_send 2; keys are read through zkeys and tkeys;
- every config uses the "main" chain: creating a wallet on the "test"
chain panics in base58_pubkey_address, so LightClient::unconnected
(which hard-codes "test") is no longer used;
- every LightClient lives in a temporary data dir, so a test cannot
write to the real ~/.silentdragonxlite or ~/.zcash-params;
- ten send_to_address recipient lists garbled by an upstream change in
2020 are restored from the upstream history.
Expectations that contradicted deliberate current behaviour are updated;
none is ignored:
- a pending send's mempool entry sits at the tip, not tip+1, and
zbalance no longer counts a note with a pending spend;
- keys cannot be added to an encrypted wallet, so the encrypted-receive
tests derive their second address before encrypting;
- the t-address vectors are m/44'/141'/0'/0/{0,1}, checked against an
independent BIP32 derivation; the z-address vectors record the current
output;
- test_addresses and test_checkpoints follow the current address count
and checkpoint tables.
All 43 tests pass on the unchanged production code (three runs) and
leave the real wallet and params directories untouched; that list is the
regression baseline for the following commits. They cover balances,
address derivation, bookkeeping, the mempool, encryption, serialization
and rollback, but none verifies a proof, an anchor or a witness.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pinned librustzcash / transitive crates (notably traitobject 0.1.0) rely on
pre-1.70 trait coherence and fail to compile on newer rustc (E0119), so the backend
must build with 1.63. Add a rust-toolchain.toml in the vendored backend so rustup
auto-selects 1.63 when cargo runs there — no more manual RUSTUP_TOOLCHAIN=1.63.0.
The pin is scoped to the backend tree (repo-root cargo keeps the default toolchain).
Also symlink the pin into the prepared build root so --silentdragonxlitelib-dir
builds honor it too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Vendor the two local Rust crates that build the lite backend artifact into
third_party/silentdragonxlite/ (the qtlib C-ABI wrapper + the silentdragonxlitelib
core, with proto/res and all the lite-send fixes), and point
build-lite-backend-artifact.sh's default --backend-dir there, so the lite wallet
builds without the upstream SilentDragonXLite repo.
External build inputs are now only the Rust toolchain + git.dragonx.is + crates.io:
- the 6 librustzcash git deps point at the git.dragonx.is/DragonX/librustzcash
mirror (pinned rev acff1444), not git.hush.is;
- the Sapling params are gitignored (not committed, no Git LFS) - the build fetches
them from the git.dragonx.is/DragonX/zcash-params 'sapling-v1' release and verifies
their SHA-256 before rust-embed bakes them in (ensure_sapling_params).
For fully offline builds, cargo vendor into lib/vendor/ and add a vendored-sources
redirect (vendor/ is gitignored; the script symlinks it into the prepared dir).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DanSandClaude Opus 5.5