DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

fix(daemon): supervise a daemon restarted after a crash

Browse Source 
After an unexpected exit the monitor thread returned on its own but
stayed joinable, and start() only spawned a monitor when none was
joinable. A node restarted after a crash (no stop() in between) then ran
unwatched: a second crash was never detected, its output was never
drained, and the dead run's stdout pipe leaked into the next fork.

- start() joins a monitor that returned on its own (only when joinable,
  at the top, after the "already running" early returns), closes the
  dead run's stdout pipe (Windows: process and pipe handles), and always
  starts a new monitor for the process it launched. crash_count_ is kept.
- New lifecycle_mutex_ serializes start(), beginStop(), stop()'s flag
  writes and joins, and endStop(), so two threads can never join the same
  monitor. The monitor thread and the state callback never take it; a
  beginStop() racing a start() waits for it to return (its longest wait
  is the ~300 ms datadir-lock poll).
- start() is refused (returns false, state unchanged) while a stop is in
  progress, i.e. between beginStop() and endStop().
- The daemon tests now start the stand-in only in isolation (temp
  -datadir override, port check skipped): they no longer write the real
  ~/.hush/DRAGONX/obsidiandragon-dragonxd.pid or tail the real debug.log.
  New tests: restart after a crash is supervised, a restarted run stops
  cleanly, no fd leak over crash+restart cycles (Linux), and start() is
  refused while stopping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-12 13:22:00 -05:00
co-authored by Claude Opus 5.5
parent f6e5038043
commit 70d8b0f687
3 changed files with 269 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/daemon/embedded_daemon.cpp
+55 -11
View File
@@ -500,6 +500,8 @@ static bool isPortInUse(int port)
bool EmbeddedDaemon::start(const std::string& binary_path)
{
std::lock_guard<std::mutex> life(lifecycle_mutex_);
// Don't start if already running or starting
if (state_ == State::Starting || state_ == State::Running) {
DEBUG_LOGF("[INFO] Daemon already %s\n", state_ == State::Starting ? "starting" : "running");
@@ -511,6 +513,27 @@ bool EmbeddedDaemon::start(const std::string& binary_path)
return true;
}
// A stop session owns this object until endStop() sets Stopped; never spawn under it (its endStop would
// adopt and abandon our new pid, and it still uses stdout_fd_/process_handle_).
if (state_ == State::Stopping) {
DEBUG_LOGF("[INFO] start(): a stop is still in progress, not starting\n");
return false;
}
// A monitor that returned on its own (unexpected exit, or a requested stop that landed before stop()) is
// still joinable. Join it, then release what its run left behind. Only in this case: if a stop session
// already joined it, the fd/handle belong to that session and endStop() closes them.
if (monitor_thread_.joinable()) {
should_stop_ = true; // a monitor somehow still looping ends within one tick
monitor_thread_.join(); // normally immediate (it already returned)
#ifdef _WIN32
if (process_handle_ != nullptr) { CloseHandle(process_handle_); process_handle_ = nullptr; }
if (stdout_read_ != nullptr) { CloseHandle(stdout_read_); stdout_read_ = nullptr; }
#else
if (stdout_fd_ >= 0) { close(stdout_fd_); stdout_fd_ = -1; } // also stops it leaking into the next fork
#endif
}
should_stop_ = false;
// Check if something is already listening on the RPC port. An isolated instance (migrate-to-
// seed) runs on its own non-default port alongside the main daemon, so it skips this bail.
int rpc_port = std::atoi(DRAGONX_DEFAULT_RPC_PORT);
@@ -691,11 +714,12 @@ bool EmbeddedDaemon::start(const std::string& binary_path)
if (pid > 0 && !rec.startToken.empty()) writeOwnershipRecord(ownershipRecordPath(), rec);
}
// Start monitor thread (if not already running)
// Supervise the process just launched. A monitor left over from an earlier run was joined above.
should_stop_ = false;
if (!monitor_thread_.joinable()) {
monitor_thread_ = std::thread(&EmbeddedDaemon::monitorProcess, this);
if (monitor_thread_.joinable()) { // unreachable under lifecycle_mutex_; never join a live monitor unsignalled
should_stop_ = true; monitor_thread_.join(); should_stop_ = false;
}
monitor_thread_ = std::thread(&EmbeddedDaemon::monitorProcess, this);
return true;
}
@@ -924,8 +948,11 @@ void EmbeddedDaemon::drainOutput()
bool EmbeddedDaemon::stop(int wait_ms)
{
stop_requested_ = true;
should_stop_ = true;
{
std::lock_guard<std::mutex> life(lifecycle_mutex_);
stop_requested_ = true;
should_stop_ = true;
}
setState(State::Stopping, "Stopping dragonxd...");
// Check if our tracked process handle is still alive
@@ -983,8 +1010,11 @@ bool EmbeddedDaemon::stop(int wait_ms)
// Final drain of debug.log
drainOutput();
if (monitor_thread_.joinable()) {
monitor_thread_.join();
{
std::lock_guard<std::mutex> life(lifecycle_mutex_);
if (monitor_thread_.joinable()) {
monitor_thread_.join();
}
}
setState(State::Stopped, "dragonxd stopped");
@@ -993,6 +1023,7 @@ bool EmbeddedDaemon::stop(int wait_ms)
bool EmbeddedDaemon::beginStop()
{
std::lock_guard<std::mutex> life(lifecycle_mutex_); // before the flags: a racing start() would reset should_stop_
stop_requested_ = true;
should_stop_ = true;
if (monitor_thread_.joinable()) {
@@ -1042,8 +1073,12 @@ EmbeddedDaemon::ResourceUsage EmbeddedDaemon::ownedResourceUsage() const
void EmbeddedDaemon::endStop(bool keepWatching)
{
std::lock_guard<std::mutex> life(lifecycle_mutex_);
if (keepWatching && isRunning()) {
// Still running and still ours: watch it again. Its exit is booked as a clean stop.
if (monitor_thread_.joinable()) { // unreachable after beginStop()'s join; never join a live monitor unsignalled
should_stop_ = true; monitor_thread_.join();
}
should_stop_ = false;
monitor_thread_ = std::thread(&EmbeddedDaemon::monitorProcess, this);
return;
@@ -1504,16 +1539,20 @@ bool EmbeddedDaemon::reapWithin(pid_t pid, int timeout_ms, int& status)
bool EmbeddedDaemon::stop(int wait_ms)
{
stop_requested_ = true;
should_stop_ = true;
bool exitedOnItsOwn = true;
// Join the monitor FIRST (it sees should_stop_ within one 100ms tick) so that from here on this is
// the only reaper. isRunning() is state-based and stays true until we set Stopped below, so it can't
// tell us the child exited: poll waitpid() on a cached pid instead. If the child exited inside that
// tick, the monitor already reaped it and, seeing stop_requested_, recorded a clean stop (pid now 0).
if (monitor_thread_.joinable()) {
monitor_thread_.join();
// lifecycle_mutex_ covers the flags and the join only, never the reap waits below.
{
std::lock_guard<std::mutex> life(lifecycle_mutex_);
stop_requested_ = true;
should_stop_ = true;
if (monitor_thread_.joinable()) {
monitor_thread_.join();
}
}
const pid_t pid = process_pid_;
@@ -1557,6 +1596,7 @@ bool EmbeddedDaemon::stop(int wait_ms)
bool EmbeddedDaemon::beginStop()
{
std::lock_guard<std::mutex> life(lifecycle_mutex_); // before the flags: a racing start() would reset should_stop_
stop_requested_ = true;
should_stop_ = true;
// The monitor sees should_stop_ within one 100ms tick. If the child exited in that window the
@@ -1649,9 +1689,13 @@ EmbeddedDaemon::ResourceUsage EmbeddedDaemon::ownedResourceUsage() const
void EmbeddedDaemon::endStop(bool keepWatching)
{
std::lock_guard<std::mutex> life(lifecycle_mutex_);
const pid_t pid = process_pid_;
if (pid > 0 && keepWatching) {
// Still running and still ours: watch it again. Its exit is booked as a clean stop.
if (monitor_thread_.joinable()) { // unreachable after beginStop()'s join; never join a live monitor unsignalled
should_stop_ = true; monitor_thread_.join();
}
should_stop_ = false;
monitor_thread_ = std::thread(&EmbeddedDaemon::monitorProcess, this);
return;
src/daemon/embedded_daemon.h
+5 -1
View File
@@ -53,7 +53,8 @@ public:
/**
* @brief Start the embedded dragonxd daemon
* @param binary_path Path to dragonxd binary (auto-detect if empty)
* @return true if process started successfully
* @return true if process started successfully (or is already running); false while a stop
* session (beginStop() .. endStop()) is still in progress
*/
bool start(const std::string& binary_path = "");
@@ -501,6 +502,9 @@ private:
std::thread monitor_thread_;
std::atomic<bool> should_stop_{false};
std::atomic<bool> stop_requested_{false}; // a stop was asked for: the next exit is not a crash
// Serializes start/beginStop/stop/endStop over monitor_thread_, should_stop_ and stop_requested_ (two
// threads joining one std::thread is undefined). Never taken by the monitor thread or the state callback.
std::mutex lifecycle_mutex_;
std::set<std::string> debug_categories_;
int max_connections_ = 0; // 0 = daemon default
bool stratum_enabled_ = false; // -stratum: host a RandomX pool (v1.3.0+; older daemons ignore it)