DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

perf(lite-backend): fetch full transactions over one shared gRPC channel

Browse Source 
At the end of a sync the wallet fetches the full transaction of each new
wallet tx, together with every other transaction in the blocks that
contain them, as decoys. Every txid got its own pool job that built a
fresh Tokio runtime (one thread per CPU), a new rustls config and a new
TCP + TLS + HTTP/2 connection for a single GetTransaction call, so a
restore or rescan opened one connection per fetched transaction. The
first error returned from the sync while the remaining jobs kept
fetching and scanning in the background, and a stalled connection, an
unparsable transaction or a panicking scan could hang the sync forever
with its lock held.

fulltx_fetch.rs now works through the same (txid, height)-sorted,
de-duplicated list on the sync's runtime, with as many workers as the
sync pool has threads (2-8, the same fan-out as before), over one gRPC
channel that is dialed lazily and re-dialed on retry. Parsing and
scanning stay on the sync pool, and the call returns once the whole
batch has finished.

Behaviour changes:
- a failure on a decoy no longer fails the sync; it fails only when one
  of the wallet's own txs could not be fetched and scanned;
- new error texts "Error reading transaction: ...", "full tx scan job
  panicked" and "fetched only X of Y wallet txs", all after the existing
  "Error fetching all txns for memos: " prefix;
- each tx gets up to 3 fetch attempts with a 120 s timeout each, where
  the old code could wait forever; a request that legitimately takes
  longer now fails;
- nothing keeps running in the background after the sync returns;
- all GetTransaction calls share one TLS/HTTP-2 connection (up to one
  dial per worker at the start, plus a re-dial per retry); the request
  list and its order are unchanged;
- if the network silently drops packets mid-batch, every attempt waits
  out its timeout until 16 consecutive failed txs stop the batch: up to
  about 12 minutes with 8 workers (48 minutes with 2) with the sync lock
  held, where the old code gave up at its first failed connection.

Which txs are requested, in what order, how often they are retried and
when the batch stops do not depend on which txs are the wallet's own;
only the final result does. A server that keeps failing one tx can see
that result through the sync's retry, but it could already tell, because
an own tx that failed is requested again on the next sync without its
decoys; nothing new leaks.

tests/fulltx_fetch.rs drives the engine with a fake fetcher: the old
request list, each item fetched and scanned once, bounded concurrency, a
request sequence independent of ownership, retries, timeouts, the
breaker, panicking and unparsable scans, and a saved wallet
byte-identical to the old serial fill. The gRPC fetcher itself is not
covered by these tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-10 09:50:03 -05:00
co-authored by Claude Opus 5.5
parent ba08cab95a
commit c233d4d503
5 changed files with 610 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files
third_party/silentdragonxlite/silentdragonxlite-cli/lib/src/lightclient.rs
Vendored
+12 -36
View File
@@ -1590,7 +1590,7 @@ pub fn start_mempool_monitor(lc: Arc<LightClient>) -> Result<(), String> {
{
let decoy_txids = all_new_txs.read().unwrap();
match self.scan_fill_fulltxs(&pool, decoy_txids.to_vec()) {
match self.scan_fill_fulltxs(&mut rt, &pool, decoy_txids.to_vec()) {
Ok(_) => Ok(object!{
"result" => "success",
"latest_block" => latest_block,
@@ -1677,50 +1677,26 @@ pub fn start_mempool_monitor(lc: Arc<LightClient>) -> Result<(), String> {
}
fn scan_fill_fulltxs(&self, pool: &ThreadPool, decoy_txids: Vec<(TxId, i32)>) -> Result<Vec<()>, String> {
fn scan_fill_fulltxs(&self, rt: &mut Runtime, pool: &ThreadPool, decoy_txids: Vec<(TxId, i32)>) -> Result<(), String> {
// We need to first copy over the Txids from the wallet struct, because
// we need to free the read lock from here (Because we'll self.wallet.txs later)
let mut txids_to_fetch: Vec<(TxId, i32)> = self.wallet.read().unwrap().txs.read().unwrap().values()
let own_txids: Vec<(TxId, i32)> = self.wallet.read().unwrap().txs.read().unwrap().values()
.filter(|wtx| wtx.full_tx_scanned == false)
.map(|wtx| (wtx.txid.clone(), wtx.block))
.collect::<Vec<(TxId, i32)>>();
info!("Fetching {} new txids, total {} with decoy", txids_to_fetch.len(), decoy_txids.len());
txids_to_fetch.extend_from_slice(&decoy_txids[..]);
txids_to_fetch.sort();
txids_to_fetch.dedup();
info!("Fetching {} new txids, total {} with decoy", own_txids.len(), decoy_txids.len());
let num_fetches = txids_to_fetch.len();
let (ctx, crx) = channel();
// Own txs and decoys go out as ONE (txid, height)-sorted, de-duplicated list, exactly as
// before, over one shared connection on the sync runtime. Request order, retries and the
// stop rule are identical for own txs and decoys, so the server cannot tell them apart.
let (items, own_set) = crate::fulltx_fetch::plan_fetch_list(own_txids, decoy_txids);
let policy = crate::fulltx_fetch::FetchPolicy::for_pool(pool);
let fetch = grpcconnector::make_shared_tx_fetcher(self.get_server_uri(), self.config.no_cert_verification);
let scan = crate::fulltx_fetch::wallet_scan_fn(self.wallet.clone());
// And go and fetch the txids, getting the full transaction, so we can
// read the memos
for (txid, height) in txids_to_fetch {
let light_wallet_clone = self.wallet.clone();
let pool = pool.clone();
let server_uri = self.get_server_uri();
let ctx = ctx.clone();
let no_cert = self.config.no_cert_verification;
pool.execute(move || {
//info!("Fetching full Tx: {}", txid);
match fetch_full_tx(&server_uri, txid, no_cert) {
Ok(tx_bytes) => {
let tx = Transaction::read(&tx_bytes[..]).unwrap();
light_wallet_clone.read().unwrap().scan_full_tx(&tx, height, 0);
ctx.send(Ok(())).unwrap();
},
Err(e) => ctx.send(Err(e)).unwrap()
};
});
};
// Wait for all the fetches to finish.
crx.iter().take(num_fetches).collect::<Result<Vec<()>, String>>()
crate::fulltx_fetch::run_fetch_scan(rt, items, own_set, policy, pool.clone(), fetch, scan)
}
pub fn do_send(&self, addrs: Vec<(&str, u64, Option<String>)>, fee: &u64) -> Result<String, String> {