DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

perf(history): write the history snapshot on change, off the UI thread

Browse Source 
Every history refresh, address refresh, connect and lock-state check
rewrote the encrypted tx-history snapshot on the UI thread: a JSON
document of every row, its encryption, then INSERT OR REPLACE and a
prune, an estimated 40-120 ms on a 5k-transaction wallet, even when
nothing had changed.

Now a signature of what would be stored (about 1 ms on 5k rows)
decides whether to write. Confirmations count only by the thresholds
the UI reads (conflicted, pending, 1-9, 10-99, 100+); per-address scan
heights, exact confirmations and the tip drift every block and form a
separate part.

- Routine refreshes write a content change at most every 20 s (a
  deferred one is released from the update loop) and the per-block
  drift at most every 10 min. Nothing changed means no write.
- A mining-address relabel, a passphrase change, a rescan, a
  disconnect, a wallet lock and exit write whatever is not on disk yet,
  at once.
- The UI thread only copies the rows and the key into a job; a single
  background writer (newest job wins) builds, encrypts and stores it. A
  failed write is retried.
- Exit queues the final write so it overlaps the daemon stop, then
  waits up to 3 s for the writer; a disk that hangs past that keeps the
  previous snapshot.
- A loaded snapshot counts as stored, so the first identical refresh
  doesn't rewrite it. An address refresh re-keys the snapshot only when
  the address set changed.

After a hard kill the snapshot can now be up to ~20 s behind for rows
and 10 min for scan heights, exact confirmations and the tip (it was
one refresh); the first refresh replaces it either way. Lite builds,
which never read this cache, no longer write it on Refresh or a mining
relabel, and the screenshot capture mode's demo rows never reach it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-09 21:09:48 -05:00
co-authored by Claude Opus 5.5
parent c0568eb930
commit 93b6871511
8 changed files with 436 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/app_network.cpp
+104 -25
View File
@@ -802,6 +802,10 @@ void App::onConnected()
void App::onDisconnected(const std::string& reason)
{
// Persist what the history snapshot is still missing while the rows, the address set (the cache identity),
// the pending-send filter and the cache key are all live: state_.clear(), pending_opids_.clear() and the
// session reset below empty them, which would turn a later flush into a silent no-op.
storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode::Now);
state_.connected = false;
state_.warming_up = false;
state_.warmup_status.clear();
@@ -1753,15 +1757,22 @@ void App::wipePendingTransactionHistoryCachePassphrase()
void App::resetTransactionHistoryCacheSession()
{
storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode::Now); // while the key is still live
transaction_history_cache_.lockKey();
wipePendingTransactionHistoryCachePassphrase();
transaction_history_cache_loaded_ = false;
history_store_ = {}; // a new session: nothing of it is known to be on disk yet
history_store_pending_ = false;
invalidateShieldedHistoryScanProgress(false);
history_listed_z_.clear();
}
void App::lockTransactionHistoryCacheSession()
{
// Write what is not on disk yet while the key is live. The queued job carries its own key copy, so the
// lockKey() below cannot lose it. history_store_ is kept: the snapshot stays valid, and re-unlocking with
// the same passphrase gives the same keyId(), so the unlock does not rewrite it.
storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode::Now);
transaction_history_cache_.lockKey();
wipePendingTransactionHistoryCachePassphrase();
transaction_history_cache_loaded_ = false;
@@ -1801,7 +1812,7 @@ void App::invalidateShieldedHistoryScanProgress(bool persistCache, bool keepSent
history_full_sweep_retry_at_ = 0;
++history_sent_epoch_;
}
if (persistCache) storeTransactionHistoryCacheIfAvailable();
if (persistCache) storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode::Now); // the cleared heights must reach disk
}
std::vector<std::string> App::shieldedAddressList() const
@@ -1919,38 +1930,104 @@ void App::loadTransactionHistoryCacheIfAvailable()
}
confirmed_cache_block_ = loaded.tipHeight;
transaction_history_cache_loaded_ = true;
// What is on disk is exactly what was just loaded: the first identical refresh must not rewrite it.
history_store_.sig = data::historySnapshotSignature(
state_.transactions, {}, data::TransactionHistoryCache::walletIdentityHash(walletIdentity),
transaction_history_cache_.keyId(), shielded_history_scan_heights_, loaded.tipHeight, loaded.tipHash);
history_store_.have = true;
history_store_.lastSubmit = std::chrono::steady_clock::now();
history_store_pending_ = false;
markTransactionsDirty();
network_refresh_.markDue(services::NetworkRefreshService::Timer::Transactions);
}
void App::storeTransactionHistoryCacheIfAvailable()
void App::storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode mode)
{
if (state_.transactions.empty()) return;
// Re-armed below only while a content change is still owed, so the pump can never spin on it.
history_store_pending_ = false;
// Lite never reads this cache (every load path is full-node RPC); the screenshot sweep's demo rows must
// never reach it.
if (isLiteBuild() || capture_mode_ || state_.transactions.empty()) return;
std::string walletIdentity = transactionHistoryCacheWalletIdentity();
if (walletIdentity.empty()) return;
try {
std::string walletIdentity = transactionHistoryCacheWalletIdentity();
if (walletIdentity.empty()) return;
if (!ensureTransactionHistoryCacheUnlockedFor(walletIdentity)) return;
pruneShieldedHistoryScanProgress();
if (!ensureTransactionHistoryCacheUnlockedFor(walletIdentity)) return;
pruneShieldedHistoryScanProgress();
std::unordered_set<std::string> pendingOpids(pending_opids_.begin(), pending_opids_.end());
std::vector<TransactionInfo> cacheTransactions;
cacheTransactions.reserve(state_.transactions.size());
for (const auto& transaction : state_.transactions) {
if (pendingOpids.find(transaction.txid) != pendingOpids.end()) continue;
cacheTransactions.push_back(transaction);
// What would be persisted: every row except the pending sends' provisional ones.
const std::unordered_set<std::string> pendingOpids(pending_opids_.begin(), pending_opids_.end());
const auto sig = data::historySnapshotSignature(
state_.transactions, pendingOpids,
data::TransactionHistoryCache::walletIdentityHash(walletIdentity),
transaction_history_cache_.keyId(), shielded_history_scan_heights_,
state_.sync.blocks, state_.sync.best_blockhash);
if (sig.rows == 0) return;
const auto now = std::chrono::steady_clock::now();
data::HistoryStoreInput in;
in.mode = mode;
in.haveStored = history_store_.have;
in.stored = history_store_.sig;
in.current = sig;
in.sinceLastSubmitSec = history_store_.lastSubmit == std::chrono::steady_clock::time_point{}
? 1e9
: std::chrono::duration<double>(now - history_store_.lastSubmit).count();
switch (data::decideHistoryStore(in)) {
case data::HistoryStoreAction::Skip:
return;
case data::HistoryStoreAction::Defer:
history_store_pending_ = true; // pumpTransactionHistoryStore() releases it from update()
return;
case data::HistoryStoreAction::Store:
break;
}
// A self-contained job: the writer thread never touches App state.
auto job = std::make_shared<data::TransactionHistoryCache::SnapshotJob>();
if (!transaction_history_cache_.captureKey(walletIdentity, job->key)) return;
job->walletIdentity = std::move(walletIdentity);
job->tipHeight = state_.sync.blocks;
job->tipHash = state_.sync.best_blockhash;
job->updatedAt = state_.last_tx_update != 0
? static_cast<std::time_t>(state_.last_tx_update)
: std::time(nullptr);
job->shieldedScanHeights = shielded_history_scan_heights_;
job->transactions.reserve(sig.rows);
for (const auto& transaction : state_.transactions) {
if (pendingOpids.find(transaction.txid) != pendingOpids.end()) continue;
job->transactions.push_back(transaction);
}
auto* cache = &transaction_history_cache_; // App member: outlives every job (see app.h)
history_writer_.submit("tx-history", [cache, job]() { return cache->writeSnapshot(std::move(*job)); });
// Only once the job is queued: a throwing submit must not leave the session believing it is on disk.
history_store_.have = true;
history_store_.sig = sig;
history_store_.lastSubmit = now;
} catch (const std::exception& e) {
// Never let a cache write break its caller (beginShutdown, onDisconnected, a refresh callback).
DEBUG_LOGF("[History] snapshot store skipped: %s\n", e.what());
} catch (...) {
DEBUG_LOGF("[History] snapshot store skipped\n");
}
if (cacheTransactions.empty()) return;
}
std::time_t updatedAt = state_.last_tx_update != 0
? static_cast<std::time_t>(state_.last_tx_update)
: std::time(nullptr);
transaction_history_cache_.replace(walletIdentity,
state_.sync.blocks,
state_.sync.best_blockhash,
cacheTransactions,
updatedAt,
shielded_history_scan_heights_);
void App::pumpTransactionHistoryStore()
{
// A failed write (disk full, locked DB) is not on disk: the next store rewrites everything, paced like
// any content change.
if (history_writer_.takeFailure("tx-history")) {
history_store_.have = false;
history_store_pending_ = true;
}
if (!history_store_pending_) return;
if (history_store_.lastSubmit != std::chrono::steady_clock::time_point{} &&
std::chrono::duration<double>(std::chrono::steady_clock::now() - history_store_.lastSubmit).count()
< data::kHistoryMinStoreIntervalSec)
return; // O(1) until the deadline
storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode::Coalesce);
}
void App::refreshData()
@@ -2448,8 +2525,10 @@ void App::refreshAddressData(services::ScanTicket ticket)
last_tx_block_height_ = -1;
network_refresh_.markDue(services::NetworkRefreshService::Timer::Transactions);
}
// An address refresh changes what the snapshot holds only through the wallet identity: re-key it
// under the new one at once when the set changed (the rows are the history refresh's business).
if (state_.transactions.empty()) loadTransactionHistoryCacheIfAvailable();
else storeTransactionHistoryCacheIfAvailable();
else if (addressSetChanged) storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode::Now);
// Hints compare the notes with the history on screen: only once it's there (the cache loaded
// just above, or a pass is done). Before that every note looks missing, and the first pass scans
// every address anyway.
@@ -3807,7 +3886,7 @@ void App::setMiningAddress(const std::string& addr, bool mining)
changed = true;
}
}
if (changed) storeTransactionHistoryCacheIfAvailable();
if (changed) storeTransactionHistoryCacheIfAvailable(data::HistoryStoreMode::Now); // user relabel: persist now
}
void App::invalidateAddressValidationCache()