perf(lite-backend): drop witness history of long-spent notes

Every note keeps a list of up to 100 witnesses, one per block, so that a
reorg can roll it back; a full list takes roughly 40-90 KB of the wallet
file. The block scan kept duplicating and trimming that list for every
note that had one, spent notes included, although a spent note never
reads its history again once its spend can no longer be rolled back. So
every note the wallet ever spent kept costing memory, per-block scan
time and file size.

Once the block window is full (101 blocks), a spent note whose spending
transaction's block is below the window's first block now gets an empty
witness list. This runs before each block's witness update and once when
a wallet is loaded. Such a spend can only be undone by a reorg that pops
the whole window, and that already empties every witness list, so reorgs
behave exactly as before.

Behaviour changes:
- notes whose spend lies below a full reorg window lose their witness
  history in memory, at load and in the next save; existing large
  wallets shrink on their first save (which, like every save, copies the
  previous file to wallet.dat.bak);
- the wallet format (v9) is unchanged; old builds and SilentDragonXLite
  read an empty list as an unspendable note, which a spent note already
  is;
- new info log at load: "Dropped witness history of N notes whose spend
  is deeper than 100 blocks".

The oracle now also accepts an empty list for a spent note that was
cleared before a later partial reorg.

New witness tests compare every block's lists with a model of the old
algorithm, including reorgs across the clear point and an old file with
full lists, which loads and re-saves to the new bytes. The 1000-seed
oracle run passes (12,207 reorgs, 993 sends verified offline), and a
differential run against the unmodified backend is byte-identical apart
from exactly the cleared lists; the old build reads the new files, and
17 sends it built from them verified offline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-10 09:50:03 -05:00
co-authored by Claude Opus 5.5
parent 6de9b5cc90
commit 7f99fbb9c6
3 changed files with 792 additions and 3 deletions
@@ -442,7 +442,7 @@ impl LightWallet {
let birthday = reader.read_u64::<LittleEndian>()?;
Ok(LightWallet{
let wallet = LightWallet{
encrypted: encrypted,
unlocked: !encrypted, // When reading from disk, if wallet is encrypted, it starts off locked.
enc_seed: enc_seed,
@@ -457,7 +457,16 @@ impl LightWallet {
config: config.clone(),
birthday,
total_scan_duration: Arc::new(RwLock::new(vec![Duration::new(0, 0)])),
})
};
// Existing wallets carry up to 100 witnesses per long-spent note; drop them now so the next
// save is small. Same predicate as the per-block call in scan_block_with_pool.
let cleared = wallet.clear_old_witnesses();
if cleared > 0 {
info!("Dropped witness history of {} notes whose spend is deeper than {} blocks", cleared, MAX_REORG);
}
Ok(wallet)
}
pub fn write<W: Write>(&self, mut writer: W) -> io::Result<()> {
@@ -1753,6 +1762,64 @@ pub fn scan_full_mempool_tx(&self, tx: &Transaction, height: i32, _datetime: u64
}
}
/// The height below which a confirmed spend can no longer be rolled back, or None while the block
/// window is not full. When the window holds > MAX_REORG blocks, every retained block is at or above
/// the returned height. An invalidate_block() that removes a spender whose WalletTx.block is below it
/// therefore pops the whole window, and that already empties every note's witness list
/// (each list holds <= MAX_REORG entries, and the window holds > MAX_REORG blocks).
fn witness_clear_height(&self) -> Option<i32> {
let blks = match self.blocks.read() {
Ok(b) => b,
Err(p) => p.into_inner(),
};
if blks.len() > MAX_REORG {
blks.first().map(|b| b.height)
} else {
None
}
}
/// Drop the witness history of notes whose confirmed spender (keyed by WalletTx.block, the same
/// field invalidate_block uses to decide removal) is below `clear_below`. Returns the number cleared.
fn clear_spent_witnesses_below(txs: &mut HashMap<TxId, WalletTx>, clear_below: i32) -> usize {
let mut to_clear: Vec<(TxId, usize)> = Vec::new();
for (txid, tx) in txs.iter() {
for (i, nd) in tx.notes.iter().enumerate() {
if nd.witnesses.is_empty() {
continue;
}
if let Some(spend_txid) = nd.spent {
if let Some(spend_tx) = txs.get(&spend_txid) {
if spend_tx.block < clear_below {
to_clear.push((*txid, i));
}
}
}
}
}
for (txid, i) in &to_clear {
if let Some(tx) = txs.get_mut(txid) {
// Vec::new(), not clear(): release the buffer.
tx.notes[*i].witnesses = Vec::new();
}
}
to_clear.len()
}
/// Clear the history of long-spent notes (used right after loading a wallet).
pub fn clear_old_witnesses(&self) -> usize {
match self.witness_clear_height() {
None => 0,
Some(h) => {
let mut txs = match self.txs.write() {
Ok(t) => t,
Err(p) => p.into_inner(),
};
LightWallet::clear_spent_witnesses_below(&mut *txs, h)
}
}
}
// Invalidate all blocks including and after "at_height".
// Returns the number of blocks invalidated
pub fn invalidate_block(&self, at_height: i32) -> u64 {
@@ -2137,6 +2204,9 @@ pub fn scan_full_mempool_tx(&self, tx: &Transaction, height: i32, _datetime: u64
// These are filled in inside the block
let new_txs;
let nfs: Vec<_>;
// Read before taking the txs lock (no lock nesting). The window ends at height-1 here.
let witness_clear_below = self.witness_clear_height();
{
// Create a write lock
let mut txs = self.txs.write().unwrap();
@@ -2159,6 +2229,11 @@ pub fn scan_full_mempool_tx(&self, tx: &Transaction, height: i32, _datetime: u64
.flatten()
.collect();
// Free the witness history of notes whose spend can no longer be rolled back.
if let Some(h) = witness_clear_below {
LightWallet::clear_spent_witnesses_below(&mut *txs, h);
}
// Prepare the note witnesses for updating
for tx in txs.values_mut() {
for nd in tx.notes.iter_mut() {
@@ -2656,3 +2731,6 @@ pub mod tests;
#[cfg(test)]
mod oracle_tests;
#[cfg(test)]
mod witness_tests;