DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

feat(daemon): deliver stop reliably and never kill the node unasked

Browse Source 
Closing the wallet sent "stop" at most twice, gated on a getinfo probe
that needs the daemon's cs_main, then killed the node after 20 s (Windows)
or 30 s (POSIX) whether or not stop had arrived. A node busy with an
hour-long witness rebuild therefore never got stop, was killed mid-
rebuild, and replayed everything since its last flush on every start.

Delivery (src/daemon/stop_delivery.h, RPCClient::open/trySendStop):
- stop goes over a dedicated client with no probe, so it never waits on
  rpc_'s lock or a cs_main-bound getinfo
- every attempt is classified, and StopDeliveryPolicy decides the next:
  Accepted / 503 = delivered; Queued (reply timeout) is resent at most
  every 30 s; QueueFull / not listening are retried every 1.5 s; a
  refusal after we were heard means the node is shutting down; a 401 on
  our own node tries the next credential source
- POSIX: a group SIGTERM goes out as soon as an attempt fails, which
  bypasses the HTTP queue entirely

Waiting (DaemonController::executeDaemonStop, pure, virtual-clock
tested):
- never kills on its own. Quitting waits as long as the node needs; the
  shutdown screen offers "Close wallet — leave node running" after 3 s,
  and "Kill node (unsafe)" (confirmed) only once stop was delivered and
  the node showed no CPU/I-O progress for ~10 min outside a rebuild
- other callers (restart, rescan, reinstall, switch, ...) wait up to
  120 s, let go once the app starts quitting, and keep watching a node
  that is still running; wallet switch refuses to go ahead then
- EmbeddedDaemon gains stop-session primitives (beginStop,
  pollOwnedProcess, signalGracefulStop, forceKillOwnedProcess,
  ownedResourceUsage over the Linux process group / Windows handle,
  endStop) and only one stop session runs at a time
- the Force Quit button and its stall heuristic are gone; "Leave
  running" aborts a pending stop attempt and closes at once

Verified against the fake daemon with the real GUI: closing mid-rebuild
now delivers stop (or SIGTERM 2 ms after a queue-full refusal) and the
node exits cleanly after its rebuild, where it was SIGKILLed before. The
shutdown screen, its "Leave running" button and the idle Kill offer
(after ~11 min) were checked by screenshot. New strings are English-
only for now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-01 22:50:11 -05:00
co-authored by Claude Opus 5.5
parent b1d5806f9e
commit 5993488176
11 changed files with 1497 additions and 146 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_phase4.cpp
+543
View File
@@ -80,6 +80,8 @@
#include <filesystem>
#include <fstream>
#include <iostream>
#include <algorithm>
#include <functional>
#include <memory>
#include <new>
#include <set>
@@ -3431,6 +3433,165 @@ void testPosixDestructorLeavesProcessRunning()
fsn::remove(script, ec);
fsn::remove(pidFile, ec);
}
// executeDaemonStop over a REAL EmbeddedDaemon child: steady clock, real waitpid/signals/CPU sampling.
// No RPC here (stop attempts report NotListening), so on POSIX the group SIGTERM is the stop channel.
class RealStopRuntime final : public dragonx::daemon::DaemonController::StopRuntime {
public:
using DC = dragonx::daemon::DaemonController;
explicit RealStopRuntime(dragonx::daemon::EmbeddedDaemon& d) : d_(d), t0_(std::chrono::steady_clock::now()) {}
double detachAt = -1.0; // choose "Leave running" at this time
double maxCpuPercent = -1.0;
double nowSec() override { return secondsSince(t0_); }
bool processAlive() override { return d_.pollOwnedProcess(); }
dragonx::daemon::StopSendResult trySendStop() override { return dragonx::daemon::StopSendResult::NotListening; }
bool nextCredentials() override { return false; }
bool sendGracefulSignal() override { return d_.signalGracefulStop(); }
double cpuSeconds() override { return d_.ownedResourceUsage().cpuSeconds; }
double ioBytes() override { return -1.0; }
bool rebuildActive() override { return false; }
DC::StopUserChoice userChoice() override
{
return (detachAt >= 0 && nowSec() >= detachAt) ? DC::StopUserChoice::Detach : DC::StopUserChoice::None;
}
void offerForceKill(bool) override {}
void forceKill() override { d_.forceKillOwnedProcess(); }
void setStatus(const DC::StopStatus& s) override { maxCpuPercent = std::max(maxCpuPercent, s.cpuPercent); }
private:
dragonx::daemon::EmbeddedDaemon& d_;
std::chrono::steady_clock::time_point t0_;
};
class RealSleepTask final : public dragonx::daemon::DaemonController::LifecycleTaskContext {
public:
bool cancelled() const override { return false; }
bool shuttingDown() const override { return false; }
void sleepForMs(int ms) override { std::this_thread::sleep_for(std::chrono::milliseconds(ms)); }
};
pid_t readPidFile(const std::filesystem::path& pidFile)
{
for (int i = 0; i < 300; ++i) {
std::ifstream in(pidFile);
long v = 0;
if (in >> v && v > 0) return static_cast<pid_t>(v);
std::this_thread::sleep_for(std::chrono::milliseconds(10));
}
return 0;
}
// S1: "Leave running" on a node that ignores SIGTERM and keeps a core busy: the stop returns at the next
// tick, nothing kills it, it is still alive afterwards, and its CPU was being measured.
void testPosixStopSessionDetach()
{
using dragonx::daemon::EmbeddedDaemon;
using DC = dragonx::daemon::DaemonController;
namespace fsn = std::filesystem;
const fsn::path pidFile = fsn::temp_directory_path() / "od_fake_daemon_detach.pid";
std::error_code ec;
fsn::remove(pidFile, ec);
const auto script = writeFakeDaemonScript("od_fake_daemon_detach.sh",
"trap '' TERM\necho $$ > \"$OD_TEST_PIDFILE\"\nwhile :; do :; done\n"); // pid file = trap is set
setenv("OD_TEST_PIDFILE", pidFile.string().c_str(), 1);
EmbeddedDaemon d;
d.setSkipPortCheck(true);
EXPECT_TRUE(d.start(script.string()));
const pid_t pid = readPidFile(pidFile);
EXPECT_TRUE(pid > 0);
EXPECT_TRUE(d.beginStop());
RealStopRuntime rt(d);
rt.detachAt = 2.0;
RealSleepTask task;
DC::StopPolicy p;
p.gracefulSignal = true;
p.userChoices = true;
const auto outcome = DC::executeDaemonStop(p, rt, task);
EXPECT_TRUE(outcome == DC::StopOutcome::Detached);
EXPECT_TRUE(rt.nowSec() < 2.6);
EXPECT_TRUE(rt.maxCpuPercent > 20.0); // the busy loop was measured
d.endStop(/*keepWatching=*/false);
EXPECT_TRUE(pid > 0 && kill(pid, 0) == 0); // left running
EXPECT_FALSE(d.isRunning());
if (pid > 0) {
kill(-pid, SIGKILL);
// endStop() left a detached reaper waiting on it: the zombie disappears once it is reaped.
bool reaped = false;
for (int i = 0; i < 200 && !reaped; ++i) {
reaped = (kill(pid, 0) != 0);
if (!reaped) std::this_thread::sleep_for(std::chrono::milliseconds(10));
}
EXPECT_TRUE(reaped);
}
unsetenv("OD_TEST_PIDFILE");
fsn::remove(script, ec);
fsn::remove(pidFile, ec);
}
// S1/S2 on POSIX: with no RPC answer, the group SIGTERM stops a node that handles it, promptly and
// without counting a crash.
void testPosixStopSessionSigterm()
{
using dragonx::daemon::EmbeddedDaemon;
using DC = dragonx::daemon::DaemonController;
const auto script = writeFakeDaemonScript("od_fake_daemon_session_term.sh",
"trap 'exit 0' TERM\nwhile :; do sleep 0.1; done\n");
EmbeddedDaemon d;
d.setSkipPortCheck(true);
EXPECT_TRUE(d.start(script.string()));
EXPECT_TRUE(d.beginStop());
RealStopRuntime rt(d);
RealSleepTask task;
DC::StopPolicy p;
p.gracefulSignal = true;
p.userChoices = true;
EXPECT_TRUE(DC::executeDaemonStop(p, rt, task) == DC::StopOutcome::Exited);
EXPECT_TRUE(rt.nowSec() < 2.0);
d.endStop(/*keepWatching=*/false);
EXPECT_TRUE(d.getState() == EmbeddedDaemon::State::Stopped);
EXPECT_EQ(d.getCrashCount(), 0);
std::error_code ec; std::filesystem::remove(script, ec);
}
// S1: a bounded wait that gives up keeps watching the node: its later exit is still seen, and booked
// as the clean stop we asked for (not a crash).
void testPosixStopSessionGiveUpKeepsWatching()
{
using dragonx::daemon::EmbeddedDaemon;
using DC = dragonx::daemon::DaemonController;
namespace fsn = std::filesystem;
const fsn::path readyFile = fsn::temp_directory_path() / "od_fake_daemon_session_slow.ready";
std::error_code rmEc;
fsn::remove(readyFile, rmEc);
// Signal readiness only once the trap is installed: a SIGTERM before that line would kill it.
const auto script = writeFakeDaemonScript("od_fake_daemon_session_slow.sh",
"trap '' TERM\necho $$ > \"$OD_TEST_PIDFILE\"\nsleep 1.5\nexit 0\n");
setenv("OD_TEST_PIDFILE", readyFile.string().c_str(), 1);
EmbeddedDaemon d;
d.setSkipPortCheck(true);
EXPECT_TRUE(d.start(script.string()));
EXPECT_TRUE(readPidFile(readyFile) > 0);
EXPECT_TRUE(d.beginStop());
RealStopRuntime rt(d);
RealSleepTask task;
DC::StopPolicy p;
p.gracefulSignal = true;
p.maxWaitSec = 0.5;
EXPECT_TRUE(DC::executeDaemonStop(p, rt, task) == DC::StopOutcome::StillRunning);
d.endStop(/*keepWatching=*/true);
EXPECT_TRUE(d.isRunning());
for (int i = 0; i < 400 && d.getState() != EmbeddedDaemon::State::Stopped; ++i)
std::this_thread::sleep_for(std::chrono::milliseconds(10));
EXPECT_TRUE(d.getState() == EmbeddedDaemon::State::Stopped);
EXPECT_EQ(d.getCrashCount(), 0);
d.stop(1000); // join the resumed monitor
unsetenv("OD_TEST_PIDFILE");
std::error_code ec;
fsn::remove(script, ec);
fsn::remove(readyFile, ec);
}
#endif // !_WIN32
#ifdef _WIN32
@@ -3481,6 +3642,380 @@ void testWindowsStopNeverKillsForeignDaemon()
}
#endif // _WIN32
// ---------------------------------------------------------------------------------------------------
// S2 + S1: delivering "stop" and the never-kill wait (DaemonController::executeDaemonStop), driven by a
// scripted runtime on a virtual clock.
// ---------------------------------------------------------------------------------------------------
void testStopDeliveryPolicy()
{
using dragonx::daemon::StopDeliveryPolicy;
using dragonx::daemon::StopSendResult;
using R = StopSendResult;
{ // Refused before queueing: not delivered, cheap to retry after 1.5s.
StopDeliveryPolicy p(true);
EXPECT_TRUE(p.shouldSend(0.0));
EXPECT_TRUE(p.record(R::QueueFull, 0.0) == R::QueueFull);
EXPECT_FALSE(p.shouldSend(1.0));
EXPECT_TRUE(p.shouldSend(1.5));
EXPECT_FALSE(p.delivered());
}
{ // Queued: the daemon has it; no resend for 30s (each resend takes a queue slot).
StopDeliveryPolicy p(true);
p.record(R::Queued, 0.0);
EXPECT_TRUE(p.heard());
EXPECT_FALSE(p.delivered());
EXPECT_FALSE(p.shouldSend(29.9));
EXPECT_TRUE(p.shouldSend(30.0));
// ...and a refusal after that means it unbound its listener to shut down.
EXPECT_TRUE(p.record(R::NotListening, 30.0) == R::ShuttingDown);
EXPECT_TRUE(p.delivered());
}
{ // NotListening before we were ever heard is just "not up / not bound": retry.
StopDeliveryPolicy p(true);
EXPECT_TRUE(p.record(R::NotListening, 0.0) == R::NotListening);
EXPECT_FALSE(p.delivered());
EXPECT_TRUE(p.shouldSend(1.5));
}
{ // Accepted / ShuttingDown: delivered, never resend.
StopDeliveryPolicy a(true), s(true);
a.record(R::Accepted, 0.0);
s.record(R::ShuttingDown, 0.0);
EXPECT_TRUE(a.delivered() && a.finished() && !a.shouldSend(1000.0));
EXPECT_TRUE(s.delivered() && s.finished() && !s.shouldSend(1000.0));
}
{ // Auth: terminal for a node we don't own; our own may retry with other credentials.
StopDeliveryPolicy unowned(false), owned(true);
unowned.record(R::Auth, 0.0);
EXPECT_TRUE(unowned.finished() && !unowned.delivered());
owned.record(R::Auth, 0.0);
EXPECT_FALSE(owned.finished());
owned.retryNow(0.2);
EXPECT_TRUE(owned.shouldSend(0.2));
}
using dragonx::daemon::stopSendResultFromKind;
using K = dragonx::rpc::RpcFailureKind;
EXPECT_TRUE(stopSendResultFromKind(K::None) == R::Accepted);
EXPECT_TRUE(stopSendResultFromKind(K::ReplyTimeout) == R::Queued);
EXPECT_TRUE(stopSendResultFromKind(K::QueueFull) == R::QueueFull);
EXPECT_TRUE(stopSendResultFromKind(K::NotListening) == R::NotListening);
EXPECT_TRUE(stopSendResultFromKind(K::ConnectTimeout) == R::NotListening);
EXPECT_TRUE(stopSendResultFromKind(K::ShuttingDown) == R::ShuttingDown);
EXPECT_TRUE(stopSendResultFromKind(K::Unhandled) == R::ShuttingDown);
EXPECT_TRUE(stopSendResultFromKind(K::Auth) == R::Auth);
EXPECT_TRUE(stopSendResultFromKind(K::Aborted) == R::Other);
}
void testProcStatParse()
{
using dragonx::daemon::EmbeddedDaemon;
long pgrp = 0;
unsigned long long ticks = 0;
// comm with spaces AND a ')': fields must be read after the LAST ')'. tpgid -1 must parse.
EXPECT_TRUE(EmbeddedDaemon::parseProcStatLine(
"4242 (weird) name) S 1 4240 4240 0 -1 4194560 100 0 0 0 250 50 0 0 20 0 1 0 12345 1000 10",
pgrp, ticks));
EXPECT_EQ(pgrp, 4240L);
EXPECT_EQ(ticks, 300ULL);
EXPECT_FALSE(EmbeddedDaemon::parseProcStatLine("garbage without parens", pgrp, ticks));
EXPECT_FALSE(EmbeddedDaemon::parseProcStatLine("1 (x) S 1", pgrp, ticks));
}
void testStopProgressTracker()
{
using Tracker = dragonx::daemon::DaemonController::StopProgressTracker;
{ // Flat CPU and I/O: idle time grows from the first sample.
Tracker tr;
for (int s = 0; s <= 300; ++s) tr.add(s, 10.0, 5e6);
EXPECT_NEAR(tr.idleSec(300.0), 300.0, 1e-9);
}
{ // 0.1 cores is progress (threshold 0.05).
Tracker tr;
for (int s = 0; s <= 300; ++s) tr.add(s, 0.1 * s, -1.0);
EXPECT_TRUE(tr.idleSec(300.0) < 1.0);
EXPECT_NEAR(tr.cpuPercent(), 10.0, 1e-6);
}
{ // A low-CPU flush still writing ~2 MB/min is progress too.
Tracker tr;
for (int s = 0; s <= 300; ++s) tr.add(s, 1.0, 2e6 / 60.0 * s);
EXPECT_TRUE(tr.idleSec(300.0) < 1.0);
}
{ // No readings at all: never "idle", so Kill is never offered.
Tracker tr;
for (int s = 0; s <= 3600; s += 10) tr.add(s, -1.0, -1.0);
EXPECT_NEAR(tr.idleSec(3600.0), 0.0, 1e-9);
EXPECT_TRUE(tr.cpuPercent() < 0);
}
}
void testEvaluateStopWait()
{
using DC = dragonx::daemon::DaemonController;
using D = DC::StopWaitDecision;
DC::StopPolicy shutdown; // the shutdown screen: owned, unbounded, choices
shutdown.userChoices = true;
DC::StopPolicy background; // a restart/rescan/switch: bounded, no choices
background.maxWaitSec = 120;
background.abandonOnShutdown = true;
auto sample = [] { DC::StopWaitSample s; return s; };
auto s = sample(); s.processAlive = false;
EXPECT_TRUE(DC::evaluateStopWait(shutdown, s) == D::Done);
s = sample(); s.choice = DC::StopUserChoice::Detach;
EXPECT_TRUE(DC::evaluateStopWait(shutdown, s) == D::Detach);
EXPECT_TRUE(DC::evaluateStopWait(background, s) == D::Wait); // no choices there
s = sample(); s.choice = DC::StopUserChoice::Kill; // Kill without an offer: ignored
EXPECT_TRUE(DC::evaluateStopWait(shutdown, s) == D::Wait);
s.killOffered = true;
EXPECT_TRUE(DC::evaluateStopWait(shutdown, s) == D::ForceKill);
s = sample(); s.delivered = true; s.idleSec = 600; // delivered + idle 10 min: offer
EXPECT_TRUE(DC::evaluateStopWait(shutdown, s) == D::OfferForceKill);
s.rebuildActive = true; // ...never during a rebuild
EXPECT_TRUE(DC::evaluateStopWait(shutdown, s) == D::Wait);
s = sample(); s.delivered = false; s.idleSec = 6000; // never delivered: no offer
EXPECT_TRUE(DC::evaluateStopWait(shutdown, s) == D::Wait);
s = sample(); s.elapsedSec = 120;
EXPECT_TRUE(DC::evaluateStopWait(background, s) == D::GiveUp);
EXPECT_TRUE(DC::evaluateStopWait(shutdown, s) == D::Wait); // the shutdown screen has no budget
s = sample(); s.shuttingDown = true;
EXPECT_TRUE(DC::evaluateStopWait(background, s) == D::GiveUp);
EXPECT_TRUE(DC::evaluateStopWait(shutdown, s) == D::Wait);
DC::StopPolicy unowned = shutdown;
unowned.owned = false;
s = sample(); s.delivered = true; s.idleSec = 6000; s.killOffered = true; s.choice = DC::StopUserChoice::Kill;
EXPECT_TRUE(DC::evaluateStopWait(unowned, s) == D::Wait); // never kill what we didn't launch
}
// A scripted node for executeDaemonStop, on a virtual clock (seconds) that sleeps and sends advance.
class MockStopRuntime final : public dragonx::daemon::DaemonController::StopRuntime {
public:
using DC = dragonx::daemon::DaemonController;
using R = dragonx::daemon::StopSendResult;
double clock = 0.0;
std::vector<R> sends; // results in order; the last one repeats
double exitAt = -1.0; // the process exits at this time (< 0: not on its own)
double exitAfterDelivery = -1.0; // ...or this long after stop/SIGTERM reached it
bool hasSignal = false; // POSIX-like graceful channel
int credentialsLeft = 0;
std::function<double(double)> cpuAt; // cumulative CPU seconds at t; empty = unknown
std::function<DC::StopUserChoice(double)> choiceAt;
bool rebuild = false;
std::vector<std::string> calls; // "stop", "sigterm", "creds", "offer", "withdraw", "kill"
double deliveredAt = -1.0, firstSigtermAt = -1.0, firstOfferAt = -1.0;
double lastStopAt = -1.0;
bool killed = false;
bool hitSafetyLimit = false;
DC::StopStatus lastStatus;
int count(const std::string& c) const { return (int)std::count(calls.begin(), calls.end(), c); }
double nowSec() override { return clock; }
bool processAlive() override
{
if (clock > 36000.0) { hitSafetyLimit = true; return false; } // a test bug, not an outcome
if (killed) return false;
if (exitAt >= 0 && clock >= exitAt) return false;
if (exitAfterDelivery >= 0 && deliveredAt >= 0 && clock >= deliveredAt + exitAfterDelivery) return false;
return true;
}
R trySendStop() override
{
calls.push_back("stop");
lastStopAt = clock;
const size_t n = calls.empty() ? 0 : (size_t)count("stop") - 1;
const R r = sends.empty() ? R::Accepted : sends[std::min(n, sends.size() - 1)];
clock += (r == R::Queued) ? 2.0 : 0.01; // a reply timeout costs the client timeout
if ((r == R::Accepted || r == R::ShuttingDown) && deliveredAt < 0) deliveredAt = clock;
return r;
}
bool nextCredentials() override
{
calls.push_back("creds");
return credentialsLeft-- > 0;
}
bool sendGracefulSignal() override
{
if (!hasSignal) return false;
calls.push_back("sigterm");
if (firstSigtermAt < 0) firstSigtermAt = clock;
if (deliveredAt < 0) deliveredAt = clock;
return true;
}
double cpuSeconds() override { return cpuAt ? cpuAt(clock) : -1.0; }
double ioBytes() override { return -1.0; }
bool rebuildActive() override { return rebuild; }
DC::StopUserChoice userChoice() override { return choiceAt ? choiceAt(clock) : DC::StopUserChoice::None; }
void offerForceKill(bool offered) override
{
calls.push_back(offered ? "offer" : "withdraw");
if (offered && firstOfferAt < 0) firstOfferAt = clock;
}
void forceKill() override
{
calls.push_back("kill");
killed = true;
}
void setStatus(const DC::StopStatus& s) override { lastStatus = s; }
};
class MockStopTask final : public dragonx::daemon::DaemonController::LifecycleTaskContext {
public:
explicit MockStopTask(MockStopRuntime& rt) : rt_(rt) {}
double shutdownAt = -1.0;
bool cancelled() const override { return false; }
bool shuttingDown() const override { return shutdownAt >= 0 && rt_.clock >= shutdownAt; }
void sleepForMs(int ms) override { rt_.clock += ms / 1000.0; }
private:
MockStopRuntime& rt_;
};
dragonx::daemon::DaemonController::StopPolicy shutdownStopPolicy(bool posix)
{
dragonx::daemon::DaemonController::StopPolicy p;
p.owned = true;
p.gracefulSignal = posix;
p.userChoices = true;
return p;
}
void testDaemonStopExecution()
{
using DC = dragonx::daemon::DaemonController;
using R = dragonx::daemon::StopSendResult;
using O = DC::StopOutcome;
{ // Already exited: not a single RPC call.
MockStopRuntime rt; MockStopTask task(rt);
rt.exitAt = 0.0;
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(false), rt, task) == O::Exited);
EXPECT_TRUE(rt.calls.empty());
}
{ // The 2026-09-30 12:21:44 -> 12:22:04 regression: the queue is full for a while. Retry until it
// lands (Windows-like, no signal channel), then wait for the exit. Never kill.
MockStopRuntime rt; MockStopTask task(rt);
rt.sends = {R::QueueFull, R::QueueFull, R::QueueFull, R::QueueFull, R::QueueFull, R::Accepted};
rt.exitAfterDelivery = 3.0;
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(false), rt, task) == O::Exited);
EXPECT_EQ(rt.count("stop"), 6);
EXPECT_EQ(rt.count("kill"), 0);
EXPECT_TRUE(rt.deliveredAt >= 5 * 1.5); // retries were spaced, not a tight loop
EXPECT_FALSE(rt.hitSafetyLimit);
}
{ // Never automatic: queue full for an hour, CPU busy (a rebuild with no log marker). No kill, no
// offer; the node exits on its own at 60 min.
MockStopRuntime rt; MockStopTask task(rt);
rt.sends = {R::QueueFull};
rt.exitAt = 3600.0;
rt.cpuAt = [](double t) { return 6.0 * t; };
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(false), rt, task) == O::Exited);
EXPECT_EQ(rt.count("kill"), 0);
EXPECT_EQ(rt.count("offer"), 0);
EXPECT_TRUE(rt.count("stop") > 1000); // still trying every 1.5s
EXPECT_FALSE(rt.hitSafetyLimit);
}
{ // POSIX: the first attempt is refused, so the group SIGTERM goes out at once (<= 2s), and no
// more RPC attempts are needed.
MockStopRuntime rt; MockStopTask task(rt);
rt.sends = {R::QueueFull};
rt.hasSignal = true;
rt.exitAfterDelivery = 2.0;
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(true), rt, task) == O::Exited);
EXPECT_TRUE(rt.firstSigtermAt >= 0 && rt.firstSigtermAt <= 2.0);
EXPECT_EQ(rt.count("stop"), 1);
EXPECT_EQ(rt.count("kill"), 0);
}
{ // Queued (reply timeout) is not a failure: no SIGTERM, no resend before 30s.
MockStopRuntime rt; MockStopTask task(rt);
rt.sends = {R::Queued};
rt.hasSignal = true;
rt.exitAt = 25.0;
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(true), rt, task) == O::Exited);
EXPECT_EQ(rt.count("stop"), 1);
EXPECT_EQ(rt.count("sigterm"), 0);
}
{ // Delivered, then flat CPU: Kill is OFFERED after ~10 min, and only an explicit Kill kills (once).
MockStopRuntime rt; MockStopTask task(rt);
rt.cpuAt = [](double) { return 42.0; };
rt.choiceAt = [](double t) { return t >= 900.0 ? DC::StopUserChoice::Kill : DC::StopUserChoice::None; };
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(false), rt, task) == O::Killed);
EXPECT_TRUE(rt.firstOfferAt >= 600.0 && rt.firstOfferAt < 602.0);
EXPECT_EQ(rt.count("kill"), 1);
EXPECT_TRUE(rt.clock >= 900.0); // waited for the user, didn't act on the offer itself
}
{ // ...with no answer to the offer, it keeps waiting (and never kills).
MockStopRuntime rt; MockStopTask task(rt);
rt.cpuAt = [](double) { return 42.0; };
rt.exitAt = 3600.0;
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(false), rt, task) == O::Exited);
EXPECT_EQ(rt.count("offer"), 1);
EXPECT_EQ(rt.count("kill"), 0);
}
{ // A rebuild in progress suppresses the offer even with flat readings.
MockStopRuntime rt; MockStopTask task(rt);
rt.cpuAt = [](double) { return 42.0; };
rt.rebuild = true;
rt.exitAt = 1800.0;
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(false), rt, task) == O::Exited);
EXPECT_EQ(rt.count("offer"), 0);
}
{ // "Leave running": returns at once, no kill, no further stop attempts.
MockStopRuntime rt; MockStopTask task(rt);
rt.sends = {R::QueueFull};
rt.choiceAt = [](double t) { return t >= 5.0 ? DC::StopUserChoice::Detach : DC::StopUserChoice::None; };
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(false), rt, task) == O::Detached);
EXPECT_EQ(rt.count("kill"), 0);
EXPECT_TRUE(rt.lastStopAt < 5.0);
EXPECT_TRUE(rt.clock < 5.5);
}
{ // A node we didn't launch is never killed, even with Kill chosen; a bounded wait then gives up.
MockStopRuntime rt; MockStopTask task(rt);
DC::StopPolicy p = shutdownStopPolicy(false);
p.owned = false;
p.maxWaitSec = 120.0;
rt.choiceAt = [](double) { return DC::StopUserChoice::Kill; };
EXPECT_TRUE(DC::executeDaemonStop(p, rt, task) == O::StillRunning);
EXPECT_EQ(rt.count("kill"), 0);
EXPECT_EQ(rt.count("offer"), 0);
EXPECT_TRUE(rt.clock >= 120.0 && rt.clock < 121.0);
}
{ // Background callers (restart, rescan, switch): bounded and never kill; they also let go as soon
// as the app starts shutting down, so the shutdown can take over.
DC::StopPolicy bg;
bg.maxWaitSec = 120.0;
bg.abandonOnShutdown = true;
MockStopRuntime a; MockStopTask ta(a);
a.cpuAt = [](double) { return 1.0; };
EXPECT_TRUE(DC::executeDaemonStop(bg, a, ta) == O::StillRunning);
EXPECT_EQ(a.count("kill"), 0);
EXPECT_TRUE(a.clock >= 120.0 && a.clock < 121.0);
MockStopRuntime b; MockStopTask tb(b);
tb.shutdownAt = 10.0;
EXPECT_TRUE(DC::executeDaemonStop(bg, b, tb) == O::StillRunning);
EXPECT_TRUE(b.clock >= 10.0 && b.clock < 11.0);
}
{ // Our own node refusing our credentials: try the next source, then deliver.
MockStopRuntime rt; MockStopTask task(rt);
rt.sends = {R::Auth, R::Auth, R::Accepted};
rt.credentialsLeft = 2;
rt.exitAfterDelivery = 1.0;
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(false), rt, task) == O::Exited);
EXPECT_EQ(rt.count("creds"), 2);
EXPECT_EQ(rt.count("stop"), 3);
}
{ // ...out of credential sources on POSIX: the graceful signal still stops it.
MockStopRuntime rt; MockStopTask task(rt);
rt.sends = {R::Auth};
rt.hasSignal = true;
rt.exitAfterDelivery = 1.0;
EXPECT_TRUE(DC::executeDaemonStop(shutdownStopPolicy(true), rt, task) == O::Exited);
EXPECT_EQ(rt.count("sigterm"), 1);
EXPECT_EQ(rt.count("kill"), 0);
}
}
// S3: on Linux a node in its final shutdown phase reads comm "hush-shutoff"; its executable still says
// dragonxd, so it must still count as running (it holds the datadir lock until it exits).
void testDaemonProcessIdentity()
@@ -8064,6 +8599,11 @@ int main()
testDaemonShutdownPolicy();
testDatadirLockGate();
testDaemonProcessIdentity();
testStopDeliveryPolicy();
testProcStatParse();
testStopProgressTracker();
testEvaluateStopWait();
testDaemonStopExecution();
#ifdef _WIN32
testWindowsStopNeverKillsForeignDaemon();
#endif
@@ -8074,6 +8614,9 @@ int main()
testPosixRequestedStopIsNotACrash();
testPosixStopSigtermIsGraceful();
testPosixDestructorLeavesProcessRunning();
testPosixStopSessionDetach();
testPosixStopSessionSigterm();
testPosixStopSessionGiveUpKeepsWatching();
#endif
testPlatformEnsureDirectory();
testVerifySaplingParams();