DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

perf(wallets): walk added wallet folders off the UI thread

Browse Source 
Every open of the Wallets dialog, Refresh, folder add or remove, and Open
on a stale row walked each user-added folder recursively on the UI thread
before the dialog drew: up to 40,000 entries per folder, roughly
50-150 ms warm on Linux and seconds on a cold disk, a network share or
Windows.

The datadir listing (a few top-level files) stays synchronous, so a
wallet just created shows at once. The recursive walk runs on a detached
worker with the same depth, hit and entry limits; its rows are merged at
the start of a later frame and only the new rows are probed.

- The dialog opens at once with the datadir rows. The external rows
  arrive when the walk ends, with "Scanning folders..." beside the
  scanned-folders caption, and the card grows once.
- Reopening within 60 s with the same folders reuses the last walk
  without touching the disk, so its rows and sizes can be up to a minute
  stale: a wallet copied into a folder in that minute needs Refresh.
  Refresh and Open on a stale row always walk again.
- While a walk or a wallet probe runs (up to 30 s) the dialog keeps
  frames coming, so rows and badges appear without moving the mouse.
- The 200-row cap now holds across all folders; each extra folder could
  add one row past it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-09 21:09:48 -05:00
co-authored by Claude Opus 5.5
parent 1658950386
commit a700d2c7ee
12 changed files with 455 additions and 144 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ui/windows/wallets_dialog.h
+182 -144
View File
@@ -11,6 +11,7 @@
#include <algorithm>
#include <atomic>
#include <chrono>
#include <cctype>
#include <cstddef>
#include <cstdint>
@@ -32,6 +33,7 @@
#include "../../util/platform.h"
#include "../../util/text_format.h"
#include "../../util/wallet_file_probe.h"
#include "../../util/wallet_folder_scan.h"
#include "../../embedded/IconsMaterialDesign.h"
#include "../notifications.h"
#include "../layout.h"
@@ -43,6 +45,17 @@
namespace dragonx {
namespace ui {
namespace wallets_dialog_detail {
// The last external-folder walk (WalletsDialog::s_ext). At namespace scope rather than nested: a static inline
// member whose nested type has default member initializers is ill-formed inside the enclosing class.
struct ExtCache {
bool valid = false;
std::string key;
std::vector<util::ScannedWallet> rows;
std::chrono::steady_clock::time_point at{};
};
} // namespace wallets_dialog_detail
class WalletsDialog {
public:
static void show(App* app) {
@@ -63,7 +76,16 @@ public:
const float dp = Layout::dpiScale();
ImGuiStyle& style = ImGui::GetStyle();
if (s_needScan) { scan(); s_needScan = false; }
if (s_needScan) { scan(s_forceScan); s_needScan = false; s_forceScan = false; }
pumpScan(); // merge a finished folder walk HERE, before anything below takes references into s_rows
// Work on other threads (folder walk, wallet-file probes): keep frames coming (~10 fps, visible window)
// so rows and badges land without the user moving the mouse. Only for kBgFrameWindowSec after the last
// walk/probe started: one stuck on a dead network mount must not keep the dialog rendering forever (a
// late result is still merged on the next frame any input produces).
constexpr double kBgFrameWindowSec = 30.0;
if ((s_scanning || (s_probe && s_probe->pending.load(std::memory_order_relaxed) > 0)) &&
std::chrono::duration<double>(std::chrono::steady_clock::now() - s_bgWorkSince).count() < kBgFrameWindowSec)
RequestProgressFrame();
// ---- Content-sized card: a stack of wallet cards, sized to the wallet count (up to
// kMaxVisibleRows, then the list scrolls). Computed from the KNOWN row count so the
@@ -497,6 +519,10 @@ public:
if (!scanFolders.empty()) {
ImGui::Dummy(ImVec2(0, Layout::spacingXs()));
Type().textColored(TypeStyle::Caption, OnSurfaceMedium(), TR("wallets_scanned_folders"));
if (s_scanning) {
ImGui::SameLine();
Type().textColored(TypeStyle::Caption, OnSurfaceDisabled(), TR("wallets_scanning"));
}
ImFont* pf = Type().caption();
const float bh2 = ImGui::GetFrameHeight();
// Front-elide a long path so its identifying tail (the leaf folder) stays visible.
@@ -544,8 +570,10 @@ public:
rst.hoverBg = StateHover();
rst.bgRounding = 4.0f * dp;
rst.tooltip = TR("refresh");
if (IconButton("##walletsRefresh", ICON_MD_REFRESH, Type().iconSmall(), ImVec2(bh, bh), rst))
if (IconButton("##walletsRefresh", ICON_MD_REFRESH, Type().iconSmall(), ImVec2(bh, bh), rst)) {
s_needScan = true;
s_forceScan = true; // explicit Refresh bypasses the 60 s folder-walk cache
}
}
ImGui::SameLine();
if (TactileButton(TR("wallets_reveal"), ImVec2(150.0f * dp, 0)))
@@ -558,15 +586,7 @@ public:
}
private:
struct WalletRow {
std::string fileName;
std::string dir;
std::string canonPath; // symlink-resolved, absolute path — the STABLE identity of the physical
// file (two paths reaching the same wallet share it), used for the
// in-place link name + to de-dup rows. Computed once at scan time.
bool inDatadir = false;
long long sizeBytes = 0;
};
using WalletRow = util::ScannedWallet; // fileName, dir, canonPath, inDatadir, sizeBytes (see the header)
// Light facts read off each wallet.dat WITHOUT loading it (util/wallet_file_probe.h), computed on a
// background thread so scan() never blocks the UI on a big file read. Results live in a shared,
@@ -585,9 +605,21 @@ private:
};
struct ProbeBatch {
std::mutex mtx;
std::vector<ProbeResult> results; // index-aligned with s_rows at the moment of scan()
std::vector<ProbeResult> results; // index-aligned with s_rows (rows are only appended after scan())
std::size_t budget = 768u * 1024u * 1024u; // shared read budget of every probe on this batch (under mtx)
std::atomic<bool> cancel{false};
std::atomic<int> pending{0}; // probe threads still running on this batch
};
// One external-folder walk. The worker fills `rows`, then publishes `ready` (release); render() merges it on
// the UI thread (pumpScan). Owned via shared_ptr by the detached worker, so it may outlive the dialog.
struct ScanBatch {
std::mutex mtx;
std::vector<util::ScannedWallet> rows;
std::atomic<bool> ready{false};
std::atomic<bool> cancel{false};
std::string key; // util::externalScanKey(folders) at launch
};
using ExtCache = wallets_dialog_detail::ExtCache;
// Force a "wallet-...dat" filename (plain, no path) so a new/imported wallet shows in the
// datadir scan and can't collide with the daemon's internal .dat files. Returns "" if the
@@ -656,8 +688,9 @@ private:
// switch to it, and the daemon guard would silently fall back to wallet.dat — "opened X, got the
// default". Refresh the list instead so the vanished row drops out — but not here: we are inside
// render()'s row loop, which holds `const WalletRow& r` and iterates s_order sized for the current
// rows, so a rebuild would leave both dangling (out-of-bounds). Flag it; the next frame rescans.
if (!fs::exists(src, ec)) { s_needScan = true; return; }
// rows, so a rebuild would leave both dangling (out-of-bounds). Flag it; the next frame rescans,
// forced so the folder-walk cache can't resurrect the ghost row.
if (!fs::exists(src, ec)) { s_needScan = true; s_forceScan = true; return; }
// canonPath (symlinks resolved) is the wallet's stable identity: hash it so the same file reached
// via any path maps to ONE link name, and point the link straight at the resolved real file.
const std::string target = r.canonPath.empty() ? canonicalOf(src) : r.canonPath;
@@ -700,149 +733,154 @@ private:
s_open = false;
}
// Scan the datadir (wallet-prefixed *.dat only, to skip peers.dat / asmap.dat / fee_estimates.dat)
// and each user-added folder (any *.dat). User-added folders are searched RECURSIVELY into
// subdirectories; the datadir stays top-level only (never descend into blocks/ chainstate/ …).
// Exception-safe: iterates with error_codes. Recursion is bounded (depth / hit / visit caps) and
// never follows directory symlinks (std default), so it can't loop or stall the UI on a huge tree.
static void scan() {
// Supersede any in-flight probe batch: its detached thread notices cancel and stops. It only ever
// touches its OWN batch (never s_rows), so rebuilding the list below is safe while it winds down.
if (s_probe) s_probe->cancel.store(true);
s_rows.clear();
namespace fs = std::filesystem;
// Guards for recursive external-folder scans (a user could point us at a large tree / home dir).
constexpr int kMaxDepth = 8; // don't descend deeper than this below the chosen folder
constexpr size_t kMaxHits = 200; // cap total wallet files listed
constexpr int kMaxVisited = 40000; // cap total entries walked, so scan() can't stall the UI
// Node/daemon .dat files that are NOT wallets. The datadir scan already restricts to a "wallet"
// prefix, but recursive external folders accept any *.dat — and a user can easily point the picker
// at a folder that IS or CONTAINS a datadir, so filter these out everywhere or the list fills with
// blk*/rev* block files, peers.dat, etc. presented as importable "wallets".
auto isNodeArtifact = [](const std::string& n) {
static const char* kExact[] = { "peers.dat", "banlist.dat", "fee_estimates.dat",
"mempool.dat", "asmap.dat", "zindex.dat" };
for (const char* e : kExact) if (n == e) return true;
return n.rfind("blk", 0) == 0 || n.rfind("rev", 0) == 0; // blk00000.dat / rev00000.dat
};
std::unordered_set<std::string> seenCanon; // one row per physical wallet. The datadir is scanned
// first, so a datadir wallet wins over the same file
// reached via an external (possibly symlinked) path.
auto consider = [&](const fs::path& p, bool inDatadir) {
// String-only checks FIRST (no syscalls) — the vast majority of entries in a large tree are
// neither .dat nor wallets, so reject them before paying for a stat().
const std::string name = p.filename().string();
if (name.size() <= 4 || name.substr(name.size() - 4) != ".dat") return;
if (inDatadir) { if (name.rfind("wallet", 0) != 0 || isLinkName(name)) return; } // datadir: wallet-prefixed, hide in-place links
else { if (isNodeArtifact(name)) return; } // external: reject node junk
std::error_code fec;
if (!fs::is_regular_file(p, fec)) return;
WalletRow r;
r.fileName = name;
r.dir = p.parent_path().string(); // the file's ACTUAL parent (may be a subdir) — Open reads r.dir + "/" + r.fileName
r.canonPath = canonicalOf(p); // symlink-resolved identity (see below)
if (!seenCanon.insert(r.canonPath).second) return; // same physical wallet already listed via another path
r.inDatadir = inDatadir;
r.sizeBytes = static_cast<long long>(fs::file_size(p, fec));
s_rows.push_back(std::move(r)); // encryption/seed flags are filled in asynchronously (see below)
};
// Directory names we never descend into during a recursive external scan — node data subtrees hold
// no user wallets and would otherwise burn the visit budget (and flood the list) with block files.
auto isNodeSubtree = [](const std::string& n) {
return n == "blocks" || n == "chainstate" || n == "database" ||
n == "indexes" || n == "index";
};
auto addFrom = [&](const std::string& dir, bool inDatadir, bool recursive) {
std::error_code ec;
if (dir.empty() || !fs::is_directory(dir, ec)) return;
if (!recursive) {
fs::directory_iterator it(dir, ec), end;
for (; !ec && it != end; it.increment(ec)) consider(it->path(), inDatadir);
return;
}
// skip_permission_denied keeps a locked subdir from aborting the whole walk; the iterator
// does NOT follow directory symlinks by default, so symlink cycles can't cause infinite loops.
fs::recursive_directory_iterator it(dir, fs::directory_options::skip_permission_denied, ec), end;
int visited = 0;
for (; !ec && it != end; it.increment(ec)) {
if (++visited > kMaxVisited) break;
std::error_code dec;
if (it->is_directory(dec)) {
// Prune: don't descend past the depth cap or into node data subtrees.
if (it.depth() >= kMaxDepth || isNodeSubtree(it->path().filename().string()))
it.disable_recursion_pending();
continue; // directories aren't wallet files
// Read light facts off each wallet.dat (encryption / seed flags / counts) on a background thread so a big
// file never stalls the UI. Captures only its own heap batch + path copies (never s_rows), so it is safe to
// outlive a re-scan or app shutdown. Results land index-aligned in the batch render() reads under the mutex.
static void runProbe(std::shared_ptr<ProbeBatch> batch,
std::vector<std::pair<std::string, std::size_t>> targets) {
// Per-file 256 MB (BDB sorts long-keyed mkey/hdchain LATE, so a small cap risks a false-negative); the
// batch's shared budget (768 MB, as before, across ALL probe threads of the batch) bounds total I/O,
// charged by bytes actually read.
constexpr std::size_t kPerFile = 256u * 1024u * 1024u;
for (const auto& t : targets) {
if (batch->cancel.load()) return;
std::size_t cap = 0;
{ std::lock_guard<std::mutex> lk(batch->mtx); cap = std::min(batch->budget, kPerFile); }
ProbeResult res;
std::size_t used = 0;
if (cap > 0) {
// Primary: an exact btree walk (encryption/seed flags + address/tx counts). If it can't fully
// parse (unknown BDB variant, or a >cap file), fall back to the cheap byte-scan for badges only.
const auto bt = util::parseWalletBtree(t.first, cap);
if (bt.parsed && bt.complete) {
res = ProbeResult{ true, true, bt.encrypted, bt.hdSeed, true, bt.addresses(), bt.txCount, bt.createdEpoch };
res.mnemonic = bt.mnemonicSeed; // read straight off the hdchain record
used = bt.bytesRead;
} else {
const auto pr = util::probeWalletFile(t.first, cap);
res = ProbeResult{ pr.isBerkeleyDB, pr.scanComplete, pr.encrypted, pr.hdSeed };
// A cap-truncated btree walk still yields DEFINITIVE positives (a found marker is
// authoritative even when the scan didn't finish) — notably the fMnemonicSeed flag;
// otherwise a large wallet probed after the budget is spent loses its seed/legacy class.
if (bt.mnemonicSeed != 0) res.mnemonic = bt.mnemonicSeed;
if (bt.hdSeed) res.hdSeed = true;
if (bt.encrypted) res.encrypted = true;
used = std::max(bt.bytesRead, pr.bytesRead);
}
consider(it->path(), inDatadir);
if (s_rows.size() >= kMaxHits) break;
}
};
addFrom(util::Platform::getDragonXDataDir(), /*inDatadir=*/true, /*recursive=*/false);
if (s_app)
for (const auto& f : s_app->walletIndex().extraFolders())
addFrom(f, /*inDatadir=*/false, /*recursive=*/true);
std::lock_guard<std::mutex> lk(batch->mtx);
batch->budget -= std::min(batch->budget, used);
if (t.second < batch->results.size()) batch->results[t.second] = res;
}
}
// Probe the encryption/seed flags on a background thread so a large wallet.dat read never stalls
// the UI. The thread is DETACHED and captures only its own heap batch + a copy of the target paths
// (no static/s_rows access), so it's safe to outlive a re-scan or app shutdown. Results land in the
// shared batch that render() reads under the mutex; badges fill in over the next few frames.
auto batch = std::make_shared<ProbeBatch>();
batch->results.resize(s_rows.size());
s_probe = batch;
// Probe s_rows[from..] into `batch` (detached; synchronous during the offline screenshot sweep so the
// captured frame is populated). Rows are only APPENDED after a scan(), so earlier indices stay valid.
static void startProbe(const std::shared_ptr<ProbeBatch>& batch, std::size_t from) {
std::vector<std::pair<std::string, std::size_t>> targets;
targets.reserve(s_rows.size());
for (std::size_t i = 0; i < s_rows.size(); ++i)
for (std::size_t i = from; i < s_rows.size(); ++i)
targets.emplace_back(s_rows[i].dir + "/" + s_rows[i].fileName, i);
auto runProbe = [](std::shared_ptr<ProbeBatch> batch,
std::vector<std::pair<std::string, std::size_t>> targets) {
// Per-file 256 MB (BDB sorts long-keyed mkey/hdchain LATE, so a small cap risks a
// false-negative); a shared budget bounds total I/O, charged by bytes actually read.
std::size_t budget = 768u * 1024u * 1024u;
constexpr std::size_t kPerFile = 256u * 1024u * 1024u;
for (const auto& t : targets) {
if (batch->cancel.load()) return;
ProbeResult res;
if (budget > 0) {
// Primary: an exact btree walk (encryption/seed flags + address/tx counts). If it
// can't fully parse (unknown BDB variant, or a >cap file), fall back to the cheap
// byte-scan for badges only (no counts).
const auto bt = util::parseWalletBtree(t.first, std::min(budget, kPerFile));
if (bt.parsed && bt.complete) {
res = ProbeResult{ true, true, bt.encrypted, bt.hdSeed, true, bt.addresses(), bt.txCount, bt.createdEpoch };
res.mnemonic = bt.mnemonicSeed; // read straight off the hdchain record
budget -= std::min(budget, bt.bytesRead);
} else {
const auto pr = util::probeWalletFile(t.first, std::min(budget, kPerFile));
res = ProbeResult{ pr.isBerkeleyDB, pr.scanComplete, pr.encrypted, pr.hdSeed };
// A cap-truncated btree walk still yields DEFINITIVE positives (a found marker is
// authoritative even when the scan didn't finish), so carry what it read — notably
// the fMnemonicSeed flag. Otherwise a large wallet probed after the shared budget is
// spent loses its seed/legacy classification and the row falls back to the (possibly
// stale) runtime badge, mislabelling a legacy wallet as a seed-phrase wallet.
if (bt.mnemonicSeed != 0) res.mnemonic = bt.mnemonicSeed;
if (bt.hdSeed) res.hdSeed = true;
if (bt.encrypted) res.encrypted = true;
budget -= std::min(budget, std::max(bt.bytesRead, pr.bytesRead));
}
}
std::lock_guard<std::mutex> lk(batch->mtx);
if (t.second < batch->results.size()) batch->results[t.second] = res;
}
if (targets.empty()) return;
auto job = [](std::shared_ptr<ProbeBatch> b, std::vector<std::pair<std::string, std::size_t>> t) {
runProbe(b, std::move(t));
b->pending.fetch_sub(1);
};
if (targets.empty()) {
// nothing to probe
} else if (s_app && s_app->isScreenshotSweeping()) {
runProbe(batch, targets); // offline sweep: probe synchronously so the captured frame is populated
batch->pending.fetch_add(1);
s_bgWorkSince = std::chrono::steady_clock::now();
if (s_app && s_app->isScreenshotSweeping()) job(batch, std::move(targets));
else std::thread(job, batch, std::move(targets)).detach();
}
// Append external rows (fresh or cached) after the datadir rows, de-duplicated against what is listed now
// (a cached walk may predate a new datadir wallet), and probe just the new ones.
static void appendExternal(std::vector<util::ScannedWallet> ext, std::unordered_set<std::string>& seen) {
const std::size_t from = s_rows.size();
for (auto& w : ext)
if (seen.insert(w.canonPath).second) s_rows.push_back(std::move(w));
if (s_rows.size() == from || !s_probe) return;
{ std::lock_guard<std::mutex> lk(s_probe->mtx); s_probe->results.resize(s_rows.size()); }
startProbe(s_probe, from);
}
// UI thread, top of render(): fold a finished walk into s_rows. All s_rows/s_probe changes happen at frame
// start (here or in scan()), never while render() holds a reference into s_rows.
static void pumpScan() {
if (!s_scan || !s_scan->ready.load(std::memory_order_acquire)) return;
std::shared_ptr<ScanBatch> sb = std::move(s_scan);
s_scan.reset();
s_scanning = false;
std::vector<util::ScannedWallet> rows;
{ std::lock_guard<std::mutex> lk(sb->mtx); rows = std::move(sb->rows); }
s_ext.valid = true;
s_ext.key = sb->key;
s_ext.rows = rows;
s_ext.at = std::chrono::steady_clock::now();
std::unordered_set<std::string> seen;
for (const auto& r : s_rows) seen.insert(r.canonPath);
appendExternal(std::move(rows), seen);
}
// Rebuild the list. The datadir pass (top-level wallet*.dat, tens of entries) stays synchronous so a
// just-created wallet shows on the next open. The RECURSIVE walk of user-added folders (bounded: depth 8,
// 200 hits, 40,000 entries per folder; never follows directory symlinks) runs on a detached worker and is
// merged by pumpScan(); a result younger than 60 s for the same folder list is reused without touching the
// disk (Refresh and a stale-row Open force a walk; add/remove folder change the key). Supersedes any
// in-flight walk/probe: those only touch their own heap batches, never s_rows.
static void scan(bool force = false) {
constexpr double kExtCacheTtlSec = 60.0;
if (s_probe) s_probe->cancel.store(true);
if (s_scan) s_scan->cancel.store(true);
s_scan.reset();
s_scanning = false;
std::unordered_set<std::string> seen; // one row per physical wallet; the datadir is first, so it wins
s_rows = util::scanDatadirWallets(util::Platform::getDragonXDataDir(), seen);
s_probe = std::make_shared<ProbeBatch>();
s_probe->results.resize(s_rows.size());
startProbe(s_probe, 0); // datadir badges start filling in at once
std::vector<std::string> folders;
if (s_app) folders = s_app->walletIndex().extraFolders();
if (folders.empty()) { s_ext = ExtCache{}; return; }
const std::string key = util::externalScanKey(folders);
const double age = s_ext.valid
? std::chrono::duration<double>(std::chrono::steady_clock::now() - s_ext.at).count() : -1.0;
if (util::externalScanCacheUsable(s_ext.valid, s_ext.key, key, age, kExtCacheTtlSec, force)) {
appendExternal(s_ext.rows, seen); // no disk walk
return;
}
auto sb = std::make_shared<ScanBatch>();
sb->key = key;
s_scan = sb;
s_scanning = true;
s_bgWorkSince = std::chrono::steady_clock::now();
auto work = [](std::shared_ptr<ScanBatch> b, std::vector<std::string> fl,
std::unordered_set<std::string> seenCopy, std::size_t already) {
util::WalletScanLimits lim; // 8 / 200 / 40000, as before
auto rows = util::scanExternalWalletFolders(fl, lim, seenCopy, already, &b->cancel);
if (b->cancel.load()) return; // superseded: publish nothing
{ std::lock_guard<std::mutex> lk(b->mtx); b->rows = std::move(rows); }
b->ready.store(true, std::memory_order_release);
};
const std::size_t listed = s_rows.size();
if (s_app && s_app->isScreenshotSweeping()) {
work(sb, std::move(folders), seen, listed); // offline sweep: synchronous, frame fully populated
pumpScan();
} else {
std::thread(runProbe, batch, targets).detach();
std::thread(work, sb, std::move(folders), std::move(seen), listed).detach();
}
}
static inline bool s_open = false;
static inline App* s_app = nullptr;
static inline bool s_needScan = false;
static inline bool s_forceScan = false; // next scan bypasses the folder-walk cache
static inline bool s_scanning = false; // an external folder walk is in flight
static inline std::shared_ptr<ScanBatch> s_scan; // that walk (null when none)
static inline ExtCache s_ext; // last walk result (60 s TTL, keyed by folder list)
static inline std::chrono::steady_clock::time_point s_bgWorkSince{}; // last walk/probe start (frame wake-ups)
static inline char s_newName[128] = "";
static inline std::vector<WalletRow> s_rows;
static inline std::shared_ptr<ProbeBatch> s_probe; // current async probe batch (index-aligned w/ s_rows)