DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

perf(startup): prepare first-run node files in the background

Browse Source 
On a first run or after an upgrade, Windows release builds wrote the
bundled Sapling params, asmap, node and miner (about 88 MB) in
App::init, before the first frame: 0.5-3 s of frozen blank window, 5-30 s
on a slow disk or under antivirus. Zip, AppImage and .app builds copied
the bundled files into the daemon dir and hashed the 51.5 MB of params on
the UI thread at the first node start, a 0.3-1.5 s freeze in the wizard
or the connect loop.

A stat-only check at startup decides whether anything is missing, stale
or unverified; a normal launch starts no thread and shows nothing.
Otherwise one background job extracts (Windows) or copies (Linux, macOS)
the files, then verifies the params once:

- The first frame paints at once. If the job takes longer than 0.4 s, a
  chip at the bottom shows "Preparing wallet files... N%", then
  "Verifying wallet files...".
- A node start on the UI thread while the job runs is deferred and
  replayed when it ends (never into a bootstrap download, a restart or a
  live connection); worker threads wait, and give up on quit. After an
  upgrade that ships a new miner, the node starts after the miner is
  written instead of before. Pool-miner starts and the miner version
  check wait too, so the old miner is never run or reported while it is
  being replaced.
- Each file goes to "<name>.partial", is flushed and renamed into place
  under one lock shared with the daemon start, the bundled-node
  reinstall and the miner extraction, and the job stops between 4 MB
  chunks at shutdown. An interrupted write used to leave a truncated
  binary that was trusted forever; a leftover .partial is now removed.

Other behaviour changes:

- Linux and macOS full-node builds copy the bundled node files (about
  77 MB) into the daemon dir at first launch, even if the wallet ends up
  on an external node; before, this happened at the first embedded start.
- Lite no longer writes the 53 MB of params and asmap it never reads.
- A disk-full or permission failure that leaves the node files unusable
  shows a warning; a miner blocked by antivirus alone does not.
- A corrupt or truncated param in a zip, AppImage or .app install is
  replaced from the bundle and verified again instead of trusted forever.
- Reinstalling the bundled node holds off the connect loop and a deferred
  start for its whole run (either could launch the old binary while the
  new one was written), and is refused while another restart is running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-09 21:09:48 -05:00
co-authored by Claude Opus 5.5
parent 47180ff840
commit 971f2abae5
19 changed files with 1245 additions and 360 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_phase4.cpp
+279
View File
@@ -30,6 +30,7 @@
#include "rpc/rpc_failure.h"
#include "rpc/rpc_busy_gate.h"
#include "resources/embedded_resources.h"
#include "resources/provision_state.h"
#include "services/history_scan_policy.h"
#include "services/wallet_scan_governor.h"
#include "services/scan_policy.h"
@@ -3341,6 +3342,278 @@ void testVerifySaplingParams()
fsn::remove_all(dir2, rmec);
}
// ── Startup node files (resources/provision_state.h, embedded_resources atomic writers, Sapling param state) ──
static std::string readProvisionFile(const fs::path& p)
{
std::ifstream f(p.string(), std::ios::binary);
return std::string((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
}
void testStartupProvisionState()
{
using namespace dragonx::resources;
EXPECT_EQ(provisionPercent(0, 0), 0);
EXPECT_EQ(provisionPercent(50, 100), 50);
EXPECT_EQ(provisionPercent(100, 100), 100);
EXPECT_EQ(provisionPercent(150, 100), 100);
EXPECT_EQ(provisionPercent(1, 3), 33);
EXPECT_TRUE(!provisionChipVisible(true, 0.1, 0.4));
EXPECT_TRUE(provisionChipVisible(true, 0.5, 0.4));
EXPECT_TRUE(!provisionChipVisible(false, 5.0, 0.4));
EXPECT_TRUE(daemonStartGate(false, true) == DaemonStartGate::Proceed);
EXPECT_TRUE(daemonStartGate(false, false) == DaemonStartGate::Proceed);
EXPECT_TRUE(daemonStartGate(true, true) == DaemonStartGate::Defer);
EXPECT_TRUE(daemonStartGate(true, false) == DaemonStartGate::Wait);
ProvisionProgress progress;
EXPECT_TRUE(progress.getPhase() == ProvisionPhase::Idle);
progress.setPhase(ProvisionPhase::Verifying);
EXPECT_TRUE(progress.getPhase() == ProvisionPhase::Verifying);
StartupProvision sp;
StartupProvision::Result r;
EXPECT_TRUE(!sp.isRunning());
EXPECT_TRUE(!sp.hasCompletion());
EXPECT_TRUE(!sp.takeCompletion(&r));
EXPECT_TRUE(sp.waitUntilDone(0));
sp.begin();
EXPECT_TRUE(sp.isRunning());
EXPECT_TRUE(!sp.hasCompletion()); // nothing to reap while it runs (no idle-loop wake for the whole job)
EXPECT_TRUE(!sp.takeCompletion(&r));
EXPECT_TRUE(!sp.waitUntilDone(10));
EXPECT_TRUE(sp.elapsedSeconds() >= 0.0);
std::thread worker([&sp] { sp.finish({ true, false, false }); });
EXPECT_TRUE(sp.waitUntilDone(5000));
worker.join();
EXPECT_TRUE(!sp.isRunning());
EXPECT_TRUE(sp.hasCompletion());
EXPECT_TRUE(sp.takeCompletion(&r));
EXPECT_TRUE(r.extractOk && !r.verifyOk && !r.cancelled);
EXPECT_TRUE(!sp.takeCompletion(&r)); // exactly once
EXPECT_TRUE(!sp.hasCompletion());
EXPECT_NEAR(sp.elapsedSeconds(), 0.0, 1e-9);
EXPECT_TRUE(!sp.endedCancelled());
// A job that shutdown cancelled: a waiting daemon start / miner start must abort, not fall through.
StartupProvision cancelledJob;
cancelledJob.begin();
EXPECT_TRUE(!cancelledJob.endedCancelled()); // still running
cancelledJob.finish({ false, true, true });
EXPECT_TRUE(cancelledJob.waitUntilDone(0));
EXPECT_TRUE(cancelledJob.endedCancelled());
EXPECT_TRUE(cancelledJob.takeCompletion(&r) && r.cancelled);
EXPECT_TRUE(cancelledJob.endedCancelled()); // a non-reaping peek: still true once reaped
StartupProvision idle;
idle.markNothingToDo();
EXPECT_TRUE(!idle.hasCompletion() && !idle.isRunning());
EXPECT_TRUE(!idle.takeCompletion(nullptr));
EXPECT_TRUE(idle.waitUntilDone(0));
EXPECT_TRUE(!idle.endedCancelled());
}
void testResourceNeedsWrite()
{
using dragonx::resources::resourceNeedsWrite;
const fs::path dir = makeTempDir();
const std::string f = (dir / "res.bin").string();
EXPECT_TRUE(resourceNeedsWrite(10, false, f)); // missing
EXPECT_TRUE(resourceNeedsWrite(10, true, f));
EXPECT_TRUE(!resourceNeedsWrite(0, true, f)); // nothing embedded
{ std::ofstream(f, std::ios::binary) << "12345"; }
EXPECT_TRUE(!resourceNeedsWrite(10, false, f)); // present, size not tracked
EXPECT_TRUE(!resourceNeedsWrite(5, true, f)); // same size
EXPECT_TRUE(resourceNeedsWrite(10, true, f)); // the bundled size differs
EXPECT_TRUE(!resourceNeedsWrite(0, true, f));
removeTree(dir);
}
void testWriteFileAtomic()
{
using namespace dragonx::resources;
const fs::path dir = makeTempDir();
std::vector<std::uint8_t> buf(9u * 1024u * 1024u); // > two 4 MB chunks
for (std::size_t i = 0; i < buf.size(); ++i) buf[i] = static_cast<std::uint8_t>((i * 31u + (i >> 13)) & 0xFFu);
const std::string expect(reinterpret_cast<const char*>(buf.data()), buf.size());
const fs::path dest = dir / "sub" / "out.bin"; // parent created on demand
const std::string partial = dest.string() + kPartialFileSuffix;
// (a) a new file: byte-exact, no temp left, progress counts every byte
{
ProvisionProgress p;
EXPECT_TRUE(writeFileAtomic(dest.string(), buf.data(), buf.size(), false, &p));
EXPECT_TRUE(readProvisionFile(dest) == expect);
EXPECT_TRUE(!fs::exists(partial));
EXPECT_EQ(p.bytesDone.load(), static_cast<std::uint64_t>(buf.size()));
}
// (b) an existing file is replaced
{
{ std::ofstream(dest.string(), std::ios::binary | std::ios::trunc) << "old contents"; }
EXPECT_TRUE(writeFileAtomic(dest.string(), buf.data(), buf.size(), false));
EXPECT_TRUE(readProvisionFile(dest) == expect);
EXPECT_TRUE(!fs::exists(partial));
}
// (c) cancelled on the 2nd poll: false, the destination is absent / unchanged, no temp left
{
int calls = 0;
const CancelFn cancel = [&calls]() { return ++calls >= 2; };
const fs::path fresh = dir / "fresh.bin";
EXPECT_TRUE(!writeFileAtomic(fresh.string(), buf.data(), buf.size(), false, nullptr, cancel));
EXPECT_TRUE(!fs::exists(fresh));
EXPECT_TRUE(!fs::exists(fresh.string() + kPartialFileSuffix));
calls = 0;
{ std::ofstream(dest.string(), std::ios::binary | std::ios::trunc) << "keep me"; }
EXPECT_TRUE(!writeFileAtomic(dest.string(), buf.data(), buf.size(), false, nullptr, cancel));
EXPECT_TRUE(readProvisionFile(dest) == "keep me");
EXPECT_TRUE(!fs::exists(partial));
}
// (d) nothing to write
{
EXPECT_TRUE(!writeFileAtomic((dir / "null.bin").string(), nullptr, 10, false));
EXPECT_TRUE(!writeFileAtomic((dir / "zero.bin").string(), buf.data(), 0, false));
EXPECT_TRUE(!fs::exists(dir / "null.bin"));
EXPECT_TRUE(!fs::exists(dir / "zero.bin"));
}
#ifndef _WIN32
// (e) executable: +x is set before the file appears under its final name
{
const fs::path exe = dir / "tool";
EXPECT_TRUE(writeFileAtomic(exe.string(), buf.data(), 1024, true));
EXPECT_TRUE((fs::status(exe).permissions() & fs::perms::owner_exec) != fs::perms::none);
}
#endif
removeTree(dir);
}
void testCopyFileAtomic()
{
using namespace dragonx::resources;
const fs::path dir = makeTempDir();
std::string body(5u * 1024u * 1024u, '\0');
for (std::size_t i = 0; i < body.size(); ++i) body[i] = static_cast<char>((i * 7u + 3u) & 0xFFu);
const fs::path src = dir / "src.bin";
{ std::ofstream out(src.string(), std::ios::binary); out.write(body.data(), static_cast<std::streamsize>(body.size())); }
#ifndef _WIN32
fs::permissions(src, fs::perms::owner_all | fs::perms::group_read | fs::perms::group_exec |
fs::perms::others_read | fs::perms::others_exec, fs::perm_options::replace); // 0755
#endif
const fs::path dst = dir / "d" / "dst.bin";
{
ProvisionProgress p;
EXPECT_TRUE(copyFileAtomic(src.string(), dst.string(), &p));
EXPECT_TRUE(readProvisionFile(dst) == body);
EXPECT_EQ(p.bytesDone.load(), static_cast<std::uint64_t>(body.size()));
EXPECT_TRUE(!fs::exists(dst.string() + kPartialFileSuffix));
#ifndef _WIN32
EXPECT_TRUE((fs::status(dst).permissions() & fs::perms::owner_exec) != fs::perms::none);
#endif
}
// Cancelled on the first poll: false, no destination, no temp.
{
const fs::path d2 = dir / "d" / "cancelled.bin";
const CancelFn cancelNow = []() { return true; };
EXPECT_TRUE(!copyFileAtomic(src.string(), d2.string(), nullptr, cancelNow));
EXPECT_TRUE(!fs::exists(d2));
EXPECT_TRUE(!fs::exists(d2.string() + kPartialFileSuffix));
}
// A missing source: false, nothing written.
{
const fs::path d3 = dir / "d" / "missing.bin";
EXPECT_TRUE(!copyFileAtomic((dir / "nope.bin").string(), d3.string()));
EXPECT_TRUE(!fs::exists(d3));
EXPECT_TRUE(!fs::exists(d3.string() + kPartialFileSuffix));
}
removeTree(dir);
}
void testBundleCopy()
{
using namespace dragonx::resources;
const fs::path root = makeTempDir();
const fs::path A = root / "A", B = root / "B", D = root / "D";
fs::create_directories(A);
fs::create_directories(B);
auto put = [](const fs::path& p, const std::string& c) {
std::ofstream(p.string(), std::ios::binary | std::ios::trunc) << c;
};
put(A / "one", "A-one");
put(B / "one", "B-one");
put(B / "two", "B-two");
const std::vector<std::string> dirs = { A.string(), B.string() };
const std::vector<std::string> names = { "one", "two", "three" }; // "three" has no source: optional
const std::string partialOne = (D / "one").string() + kPartialFileSuffix;
// The plan: the first search dir holding a name wins; names with no source are skipped.
const auto plan = planBundleCopy(dirs, D.string(), names, false);
EXPECT_EQ(plan.size(), static_cast<std::size_t>(2));
bool sawOne = false;
for (const auto& it : plan) {
if (it.name != "one") continue;
sawOne = true;
EXPECT_TRUE(fs::path(it.src) == A / "one");
EXPECT_TRUE(fs::path(it.dst) == D / "one");
}
EXPECT_TRUE(sawOne);
// A stale temp from an interrupted copy is removed by the run that copies that name.
fs::create_directories(D);
put(fs::path(partialOne), "stale");
auto r = copyBundledFiles(dirs, D.string(), names, false);
EXPECT_TRUE(r.ok);
EXPECT_EQ(r.copied, 2);
EXPECT_TRUE(r.firstFailure.empty());
EXPECT_TRUE(readProvisionFile(D / "one") == "A-one");
EXPECT_TRUE(readProvisionFile(D / "two") == "B-two");
EXPECT_TRUE(!fs::exists(D / "three"));
EXPECT_TRUE(!fs::exists(partialOne));
// Everything present: the stat-only early return copies nothing.
r = copyBundledFiles(dirs, D.string(), names, false);
EXPECT_TRUE(r.ok);
EXPECT_EQ(r.copied, 0);
EXPECT_TRUE(planBundleCopy(dirs, D.string(), names, false).empty());
// An existing destination is kept without replaceExisting and overwritten with it.
put(D / "one", "local");
r = copyBundledFiles(dirs, D.string(), { "one" }, false);
EXPECT_EQ(r.copied, 0);
EXPECT_TRUE(readProvisionFile(D / "one") == "local");
r = copyBundledFiles(dirs, D.string(), { "one" }, true);
EXPECT_TRUE(r.ok);
EXPECT_EQ(r.copied, 1);
EXPECT_TRUE(readProvisionFile(D / "one") == "A-one");
removeTree(root);
}
void testSaplingParamState()
{
using dragonx::rpc::Connection;
using State = Connection::SaplingParamState;
const fs::path dir = makeTempDir() / "sapling_state";
fs::create_directories(dir);
auto writeFile = [](const fs::path& p, const std::string& c) {
std::ofstream(p.string(), std::ios::binary | std::ios::trunc) << c;
};
const std::string spend = "fake-spend-params-contents";
const std::string output = "fake-output-params-contents";
const std::vector<std::pair<std::string, std::string>> digests = {
{ "sapling-spend.params", dragonx::util::sha256Hex(spend.data(), spend.size()) },
{ "sapling-output.params", dragonx::util::sha256Hex(output.data(), output.size()) },
};
writeFile(dir / "sapling-spend.params", spend);
EXPECT_TRUE(Connection::saplingParamStateIn(dir.string(), digests) == State::Missing); // one absent
writeFile(dir / "sapling-output.params", output);
EXPECT_TRUE(Connection::saplingParamStateIn(dir.string(), digests) == State::NeedsHash); // no marker yet
EXPECT_TRUE(Connection::verifySaplingParamsIn(dir.string(), digests));
EXPECT_TRUE(Connection::saplingParamStateIn(dir.string(), digests) == State::Verified);
writeFile(dir / "sapling-spend.params", spend + "-longer"); // size changed
EXPECT_TRUE(Connection::saplingParamStateIn(dir.string(), digests) == State::NeedsHash);
EXPECT_TRUE(Connection::saplingParamStateIn("", digests) == State::Missing);
removeTree(dir.parent_path());
}
// Wallets dialog: the pure folder walk that moved out of WalletsDialog::scan() (now run on a worker).
void testWalletFolderScan()
{
@@ -13688,6 +13961,12 @@ int main()
#endif
testPlatformEnsureDirectory();
testVerifySaplingParams();
testStartupProvisionState();
testResourceNeedsWrite();
testWriteFileAtomic();
testCopyFileAtomic();
testBundleCopy();
testSaplingParamState();
testWalletFolderScan();
testExternalScanCache();
testConnectHasStalled();