db8d5359a4b8eb422b22efea0248c053d875fcda
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  DanSandClaude Opus 5.5
|
db8d5359a4 |
fix(lite-backend): don't panic on a reorg while a send is pending
invalidate_block unwrapped a note's `spent` field for notes that only had `unconfirmed_spent` set, so any reorg while one of our sends was still in the mempool panicked the backend (and poisoned the wallet lock). The branch could only panic or do nothing; it is gone, and a reorg now leaves `unconfirmed_spent` as it is: the send can still be mined on the new branch, because the chain checks the anchor's root, not its height. The oracle no longer steers reorgs around pending sends: it now reorgs above, across and below the sent tip, including popping the anchor block, and the reproducer runs by default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DanSandClaude Opus 5.5
|
c233d4d503 |
perf(lite-backend): fetch full transactions over one shared gRPC channel
At the end of a sync the wallet fetches the full transaction of each new wallet tx, together with every other transaction in the blocks that contain them, as decoys. Every txid got its own pool job that built a fresh Tokio runtime (one thread per CPU), a new rustls config and a new TCP + TLS + HTTP/2 connection for a single GetTransaction call, so a restore or rescan opened one connection per fetched transaction. The first error returned from the sync while the remaining jobs kept fetching and scanning in the background, and a stalled connection, an unparsable transaction or a panicking scan could hang the sync forever with its lock held. fulltx_fetch.rs now works through the same (txid, height)-sorted, de-duplicated list on the sync's runtime, with as many workers as the sync pool has threads (2-8, the same fan-out as before), over one gRPC channel that is dialed lazily and re-dialed on retry. Parsing and scanning stay on the sync pool, and the call returns once the whole batch has finished. Behaviour changes: - a failure on a decoy no longer fails the sync; it fails only when one of the wallet's own txs could not be fetched and scanned; - new error texts "Error reading transaction: ...", "full tx scan job panicked" and "fetched only X of Y wallet txs", all after the existing "Error fetching all txns for memos: " prefix; - each tx gets up to 3 fetch attempts with a 120 s timeout each, where the old code could wait forever; a request that legitimately takes longer now fails; - nothing keeps running in the background after the sync returns; - all GetTransaction calls share one TLS/HTTP-2 connection (up to one dial per worker at the start, plus a re-dial per retry); the request list and its order are unchanged; - if the network silently drops packets mid-batch, every attempt waits out its timeout until 16 consecutive failed txs stop the batch: up to about 12 minutes with 8 workers (48 minutes with 2) with the sync lock held, where the old code gave up at its first failed connection. Which txs are requested, in what order, how often they are retried and when the batch stops do not depend on which txs are the wallet's own; only the final result does. A server that keeps failing one tx can see that result through the sync's retry, but it could already tell, because an own tx that failed is requested again on the next sync without its decoys; nothing new leaks. tests/fulltx_fetch.rs drives the engine with a fake fetcher: the old request list, each item fetched and scanned once, bounded concurrency, a request sequence independent of ownership, retries, timeouts, the breaker, panicking and unparsable scans, and a saved wallet byte-identical to the old serial fill. The gRPC fetcher itself is not covered by these tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DanSandClaude Opus 5.5
|
ba08cab95a |
perf(lite-backend): share unchanged witness states between blocks
Each scanned block deep-copied the newest witness of every note with history to start that block's entry, although a block without shielded outputs changes no witness, and spent or pending notes never change theirs. A note's 100-entry list was 100 separate copies, mostly identical. The list now holds Arc-shared witnesses. The per-block duplicate is an Arc::clone, and Arc::make_mut copies the newest entry only for unspent, non-pending notes and only in blocks with at least one shielded output. Every entry is still written in full, so the file is unchanged, and a spend takes its own copy of the witness. A loaded wallet is not de-duplicated, so the saving builds up over the next 100 blocks. No output change: the saved bytes are identical; notes use less memory. With 1000 live notes x 100 witnesses (a 70 MB file), RssAnon after 100 blocks without outputs is 18 MiB instead of 189 MiB and such a block scans in 1.0 ms instead of 2.1-2.7 ms; when every block has an output, the live bytes stay about the same. The witness tests, the 1000-seed oracle and a differential run against the unmodified backend (every witness entry and the whole saved file after every operation; only the cleared lists and the chosen spend notes may differ) all pass, and the measured wallet saves to the same SHA-256 as with the old build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DanSandClaude Opus 5.5
|
6f1ae874fb |
feat(lite-backend): select spend notes deterministically, max 250 inputs
send_to_address walked the wallet's notes in HashMap order and took them until the target was covered, cloning the witness and spending key of each note it visited, zero-value notes included. So which notes were spent, how many and which address got the change were random: a send could prove zero-value notes (each spend is a Groth16 proof of about 0.5-2 s plus 384 bytes) and far more notes than needed, and a send needing 251-489 notes ground through minutes of proving to build a transaction whose flat fee is below DragonX's default minimum relay rate. Selection is now a pure function (note_selection.rs) over the value and nullifier of the eligible non-zero notes: - the smallest single note that covers the target; otherwise the fewest notes, largest first, with the last pick swapped for the smallest note that still covers. Ties break on the nullifier, so map order no longer matters; - at most 250 spends, and fewer when the estimated size would pass 190,000 bytes (the consensus limit is 200,000); - the notes cover the whole target whenever they can; only when they cannot do the keyed UTXOs, which are still spent in full, reduce what the notes must cover; - only the chosen notes get their witness and key cloned. Behaviour changes: - which notes are spent, and so which address receives the change, is deterministic: the smallest single covering note, otherwise the fewest notes, largest first; - zero-value notes are never spent; - new errors before any proving: "Too many inputs: ..." (more than 250 spends needed; it states the maximum sendable amount in DRGX) and "Transaction too large: ..." (it suggests shielding transparent funds first). Neither contains "Insufficient", so the GUI does not retry them. Sends that need 251-489 notes, which succeed today after minutes of proving, are refused; - UTXOs alone fund a send only when no eligible non-zero note exists, or when the size budget leaves no room for a shielded input; then, as for a wallet without notes before, the change goes to the first z-address and is listed as a receive. Otherwise the smallest non-zero note is still spent, so the change is recorded as change; - the "Insufficient verified funds" text is unchanged, but its "have" amount no longer counts UTXOs whose key the wallet lacks; - duplicate nullifiers (corrupt state only) are de-duplicated; - new info log: "Selected N of M spendable notes". A note left with a stale witness by the pre-existing pending-send bug is now picked every time it is the best fit, rather than by chance. Pure tests check that the selection never uses more notes than the old walk (3000 random cases, shuffled), that the count is minimal (brute force) and independent of order, and the cap, size budget and fallbacks; wallet tests cover best fit, a chat-fee send, the cap refused before proving and UTXO funding. In the 1000-seed oracle every send verifies offline at anchor offsets 0 and 4 and the largest send dropped from 7 inputs to 2; a differential run against the unmodified backend has identical anchors and byte-identical non-selection state apart from the cleared lists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DanSandClaude Opus 5.5
|
965f36cecb |
perf(lite-backend): batch a block's witness appends across threads
The block scan appended each shielded output's leaf to the witness of every live (unspent, not pending) note as it went, one witness at a time on the sync thread, while holding the wallet's txs write lock: live notes x outputs appends per block, all on one core. With 16 or more live notes, the scan now collects the block's leaves in output order and appends them to those witnesses once, at the end of the block, still under the same lock and before anything reads them. When that is at least 128 appends, the witnesses are split into disjoint chunks over min(8, cores, live notes / 8) scoped threads; otherwise the appends run inline. Every witness receives the same leaves in the same order, including a leaf whose epk does not parse; an output whose cmu does not parse still adds nothing. The hashing work is the same, only spread over cores; the speed-up has not been benchmarked yet. No output change. Each qualifying block starts up to min(8, cores) short-lived threads. Tests check that the threaded helper matches the plain double loop byte for byte for 0-64 witnesses, and that the deferred path handles bad-epk and bad-cmu outputs. The 1000-seed oracle passes, as do the suite and 125 oracle seeds pinned to 2 cores, and a differential run against the unmodified backend is byte-identical, apart from the lists cleared two commits earlier, for wallets below and above 16 live notes and blocks of 0, 1 and 50 outputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DanSandClaude Opus 5.5
|
e7ebc568b0 |
perf(lite-backend): trim per-note witness history in place
The block scan replaced each note's witness list with the result of split_off(), which allocates a new buffer on every call: every block, each note with history paid an allocation and, once its list was full, a copy of all 100 entries. The duplicate of the newest witness is now pushed and, when the list exceeds 100 entries, the oldest is drained in place; notes with no history are skipped. The list keeps the same entries in the same order, so nothing in the output changes. The witness tests and the oracle pass, and a differential run against the unmodified backend is byte-identical (every witness entry and the whole saved file after every operation) apart from the lists the previous commit clears. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DanSandClaude Opus 5.5
|
7f99fbb9c6 |
perf(lite-backend): drop witness history of long-spent notes
Every note keeps a list of up to 100 witnesses, one per block, so that a reorg can roll it back; a full list takes roughly 40-90 KB of the wallet file. The block scan kept duplicating and trimming that list for every note that had one, spent notes included, although a spent note never reads its history again once its spend can no longer be rolled back. So every note the wallet ever spent kept costing memory, per-block scan time and file size. Once the block window is full (101 blocks), a spent note whose spending transaction's block is below the window's first block now gets an empty witness list. This runs before each block's witness update and once when a wallet is loaded. Such a spend can only be undone by a reorg that pops the whole window, and that already empties every witness list, so reorgs behave exactly as before. Behaviour changes: - notes whose spend lies below a full reorg window lose their witness history in memory, at load and in the next save; existing large wallets shrink on their first save (which, like every save, copies the previous file to wallet.dat.bak); - the wallet format (v9) is unchanged; old builds and SilentDragonXLite read an empty list as an unspendable note, which a spent note already is; - new info log at load: "Dropped witness history of N notes whose spend is deeper than 100 blocks". The oracle now also accepts an empty list for a spent note that was cleared before a later partial reorg. New witness tests compare every block's lists with a model of the old algorithm, including reorgs across the clear point and an old file with full lists, which loads and re-saves to the new bytes. The 1000-seed oracle run passes (12,207 reorgs, 993 sends verified offline), and a differential run against the unmodified backend is byte-identical apart from exactly the cleared lists; the old build reads the new files, and 17 sends it built from them verified offline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DanSandClaude Opus 5.5
|
6de9b5cc90 |
perf(lite-backend): borrow embedded Sapling params instead of copying
Every LightClient copied the Sapling spend and output parameters, which rust-embed already compiles into the binary, onto the heap: 51,551,256 bytes (49.2 MiB) of private memory per live client, whether or not it ever sent anything. LightClient::sapling_output and sapling_spend are now Cow<'static, [u8]> and borrow the embedded image. set_sapling_params still stores an owned copy, with its is-empty guard, hash checks and disk write unchanged. The prover receives the same bytes. Behaviour changes: - about 49 MiB less anonymous memory per live LightClient; the wallet file and JSON are unchanged; - the first send or shield pages the 51.5 MB of embedded params in from the binary, where client creation used to copy them (an estimated 0.3-1 s from the AppImage); - the public fields sapling_output and sapling_spend change type from Vec<u8> to Cow<'static, [u8]>: a Rust API change, with no C ABI or exported-symbol change. tests/sapling_params.rs checks that a restored, a new and a reloaded client all borrow the embedded image (same pointer, pinned lengths and SHA-256), that set_sapling_params keeps its guard and hash checks, and that the borrowed params parse like a heap copy. An ignored test measures 576 KiB of RssAnon growth for a second client, against 46.5 MiB with the old heap copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DanSandClaude Opus 5.5
|
bd4c06dfd0 |
perf(lite-backend): encode note addresses from pk_d instead of deriving
note_address() re-derived a note's payment address from its viewing key and diversifier (a group hash, an ivk hash and a 252-bit scalar multiplication) for every note in `notes` and `list` and for each new note in the block scan. The z-balance sums did the same derivation in their address filter, so `balance`, which sums four balances for each of Z z-addresses, cost Z x (N + 2U) derivations for N notes, U of them unspent. do_list_notes also encoded up to Z + Z^2 addresses to find the spendable ones and cloned that set once per transaction. Now: - note_address() encodes the note's stored pk_d with its diversifier. Both decryption and wallet load build that pk_d from the same key and diversifier, so the string is byte-identical; - the three z-balance sums share one address filter. The per-address sums behind `balance` compare (diversifier, pk_d) with each z-address directly; any other address string keeps the old exact string compare; - those sums work from a copy of the z-addresses whose lock is released before any note is read, so the txs-then-zkeys lock order that sync uses is unchanged; - do_list_notes builds the spendable-address set with Z encodes and borrows it. No output changes: balances and the notes/list/balance JSON are byte-identical. The address filter no longer contains an unwrap (it could not fail). Tests compare against verbatim copies of the old functions over randomized chains (note addresses, also after a reload; every filtered balance for own, diversified, foreign and junk address strings; the per-address balances; the spendable set), and a concurrency test fails with a deadlock if the zkeys lock is held across the sums. A differential run against the unmodified backend compared per-address balances and the JSON after every operation, through reorgs and reloads, and the old build re-saves the new files byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DanSandClaude Opus 5.5
|
d28cc0da31 |
test(lite-backend): add a seeded chain oracle and offline spend verifier
No existing test checks that a send's anchor, nullifiers and proofs are valid, or that note witnesses stay right through reorgs and reloads, and that is what the following witness and note-selection changes touch. oracle_tests.rs drives a LightWallet through its public entry points over seeded random fake chains that start at real mainnet checkpoints: receives to HD, diversified, imported, view-only and foreign keys, zero-value and dust notes, outputs with an unparseable epk, compact spends with change, real sends that are mined later or never (mempool expiry), reorgs of 1-100 blocks in one call or block by block (re-mining spends on the new branch), and save/reload. An independent reference keeps its own commitment tree, every block's root and every note's true witness at each height. After each step it checks the tree and block window, every note's fields and spent state, witness-list lengths, every retained witness of a live note byte for byte (sampled ones also by root, position and auth path), the balances and the mempool. Sends are captured in the broadcast closure, never sent, and verified offline: the anchor must be the reference root at tip minus the anchor offset, every nullifier must belong to an eligible note, and the Sapling verifier must accept every spend and output proof and the binding signature (the first send of a run also checks that it rejects a wrong anchor and a wrong nullifier). The default run (20 seeds x 300 blocks) is part of the normal suite; a 1000-seed x 400-block run and a 20-seed x 5000-block soak are #[ignore]. Every run prints its seed, and a single seed can be re-run. Two pre-existing bugs are documented as #[ignore] reproducers and kept out of the green runs: - a reorg while a send is pending panics the backend (invalidate_block unwraps `spent` on a note that only has `unconfirmed_spent`); - a pending send's notes stop receiving witness updates, so after a restart (the send never mined) or a reorg that undoes the block that mined it, they are spendable again with stale witnesses, and later sends from them fail until a rescan. On the unchanged production code the default and full runs pass (1000 seeds: 12,194 reorgs up to 100 deep, 5,893 reloads, 993 proving sends verified offline at anchor offsets 0 and 4) and the reproducers fail as described. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DanSandClaude Opus 5.5
|
8895038c40 |
test(lite-backend): port the core-crate tests to the current API
The lite backend core's in-crate tests no longer compiled (119 errors in
lightwallet/tests.rs and the lightclient and walletzkey test modules):
they still used the old LightWallet::new and send_to_address signatures,
the removed extfvks/extsks/zaddress fields and the old new_from_phrase
and do_send arities. So `cargo test` built nothing and no backend change
could be checked against them.
Port them, and the commands and checkpoints tests, to the current API,
touching only #[cfg(test)] code:
- LightWallet::new takes 4 arguments, send_to_address 7 (with an
explicit default fee, returning (txid, raw tx)), new_from_phrase 5 and
do_send 2; keys are read through zkeys and tkeys;
- every config uses the "main" chain: creating a wallet on the "test"
chain panics in base58_pubkey_address, so LightClient::unconnected
(which hard-codes "test") is no longer used;
- every LightClient lives in a temporary data dir, so a test cannot
write to the real ~/.silentdragonxlite or ~/.zcash-params;
- ten send_to_address recipient lists garbled by an upstream change in
2020 are restored from the upstream history.
Expectations that contradicted deliberate current behaviour are updated;
none is ignored:
- a pending send's mempool entry sits at the tip, not tip+1, and
zbalance no longer counts a note with a pending spend;
- keys cannot be added to an encrypted wallet, so the encrypted-receive
tests derive their second address before encrypting;
- the t-address vectors are m/44'/141'/0'/0/{0,1}, checked against an
independent BIP32 derivation; the z-address vectors record the current
output;
- test_addresses and test_checkpoints follow the current address count
and checkpoint tables.
All 43 tests pass on the unchanged production code (three runs) and
leave the real wallet and params directories untouched; that list is the
regression baseline for the following commits. They cover balances,
address derivation, bookkeeping, the mempool, encryption, serialization
and rollback, but none verifies a proof, an anchor or a witness.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DanSandClaude Opus 4.8
|
fffee9f0b5 |
build(lite-backend): pin the SDXL backend to rustc 1.63 via rust-toolchain.toml
The pinned librustzcash / transitive crates (notably traitobject 0.1.0) rely on pre-1.70 trait coherence and fail to compile on newer rustc (E0119), so the backend must build with 1.63. Add a rust-toolchain.toml in the vendored backend so rustup auto-selects 1.63 when cargo runs there — no more manual RUSTUP_TOOLCHAIN=1.63.0. The pin is scoped to the backend tree (repo-root cargo keeps the default toolchain). Also symlink the pin into the prepared build root so --silentdragonxlitelib-dir builds honor it too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
DanSandClaude Opus 4.8
|
1393d9ae1a |
feat(lite): vendor SDXL backend source; source deps from git.dragonx.is
Vendor the two local Rust crates that build the lite backend artifact into third_party/silentdragonxlite/ (the qtlib C-ABI wrapper + the silentdragonxlitelib core, with proto/res and all the lite-send fixes), and point build-lite-backend-artifact.sh's default --backend-dir there, so the lite wallet builds without the upstream SilentDragonXLite repo. External build inputs are now only the Rust toolchain + git.dragonx.is + crates.io: - the 6 librustzcash git deps point at the git.dragonx.is/DragonX/librustzcash mirror (pinned rev acff1444), not git.hush.is; - the Sapling params are gitignored (not committed, no Git LFS) - the build fetches them from the git.dragonx.is/DragonX/zcash-params 'sapling-v1' release and verifies their SHA-256 before rust-embed bakes them in (ensure_sapling_params). For fully offline builds, cargo vendor into lib/vendor/ and add a vendored-sources redirect (vendor/ is gitignored; the script symlinks it into the prepared dir). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |