DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

feat(rpc): stop flooding a busy node's RPC queue; fix the reconnect storm

Browse Source 
dragonxd serves RPC with 8 workers and a 16-deep queue. While a witness
rebuild holds cs_main, every request needing it waits, and a request the
client gave up on keeps its slot. The GUI kept polling and re-probing
until all 24 slots were its own abandoned requests (even stop was then
refused), treated the busy node as lost (tearing down the wallet state),
and retried connect at frame rate on some pages.

Busy gate (src/rpc/rpc_busy_gate.h), enforced in RPCClient::performCall
for the shared rpc_ / fast_rpc_:
- the first reply timeout or queue-full refusal marks the node busy;
  from then on nothing needing cs_main is sent (calls fail at once with
  the new RpcFailureKind::Deferred). stop, help and getrescaninfo are
  lock-free in dragonxd and always go through
- one canary getinfo with no client timeout waits on its own client and
  thread; its answer clears the gate and triggers an immediate refresh
  (queue-full: retried after 2..30 s; no answer: back to normal handling)
- the timer dispatch skips Core/Tx/Addresses/Peers/chat/mining/opid and
  connect attempts while busy
- the 3-strike teardown counts only failures that put the connection in
  doubt, never busy ones; no balance scans follow a busy chain-info call

Reconnect storm: retries were armed with setTimer(Core, kCoreDefault - 1),
i.e. "4 s elapsed", which against Overview's 2 s interval meant every
frame. RefreshScheduler::deferFor() is an absolute hold, and
ConnectRetryBackoff picks the delay by failure kind (not listening
1/2/5 s, queue full 2..30 s, busy 30 s, else 4 s).

Measured against the fake daemon: abandoned requests during a rebuild
went from ~24 (every slot) to at most 2, with no teardown and recovery
when the rebuild ends; a saturated queue sees about one canary per
back-off step instead of ~190 requests/s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-02 09:49:55 -05:00
co-authored by Claude Opus 5.5
parent 0c821333c8
commit 0f25c67600
12 changed files with 501 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/app_network.cpp
+91 -15
View File
@@ -264,6 +264,79 @@ void App::detectWalletDegraded()
VERBOSE_LOGF("[recovery] Daemon opened wallet in DEGRADED mode — new-key derivation disabled\n");
}
// One canary per busy period: a getinfo on its own client with NO client timeout, so it is the one request
// we let wait in the node's queue instead of re-sending (each re-send would take another slot). Its answer
// means the node serves cs_main requests again.
enum CanaryState { kCanaryIdle = 0, kCanaryRunning, kCanaryAnswered, kCanaryRefused, kCanaryGone };
void App::abortRpcBusyCanary()
{
std::lock_guard<std::mutex> lk(canary_mutex_);
if (canary_client_) canary_client_->requestAbort();
}
void App::pumpRpcBusyCanary()
{
const int st = canary_state_.load();
if (st >= kCanaryAnswered) {
if (canary_thread_.joinable()) canary_thread_.join();
canary_state_ = kCanaryIdle;
if (st == kCanaryAnswered) {
DEBUG_LOGF("[busy] the node answered again after %.0fs; resuming normal polling\n",
rpc_busy_gate_->busySeconds());
rpc_busy_gate_->clear();
canary_backoff_.reset();
connect_backoff_.reset();
network_refresh_.markImmediateRefresh(); // also makes a pending connect attempt due at once
} else if (st == kCanaryRefused) {
const float wait = canary_backoff_.next(rpc::RpcFailureKind::QueueFull);
canary_retry_at_ = std::chrono::steady_clock::now() +
std::chrono::milliseconds(static_cast<int>(wait * 1000.0f));
DEBUG_LOGF("[busy] canary refused (queue full); next one in %.0fs\n", wait);
} else {
// Nothing listening any more (or our own abort): there's no busy node to wait for. Normal
// connect / lost-connection handling takes over.
DEBUG_LOGF("[busy] canary got no answer; leaving busy mode\n");
rpc_busy_gate_->clear();
canary_backoff_.reset();
}
}
if (!rpc_busy_gate_->busy() || canary_state_.load() != kCanaryIdle || shutting_down_) return;
if (std::chrono::steady_clock::now() < canary_retry_at_) return;
const rpc::ConnectionConfig cfg = (!saved_config_.rpcuser.empty() || !saved_config_.rpcpassword.empty())
? saved_config_ : rpc::Connection::autoDetectConfig();
DEBUG_LOGF("[busy] the node is too busy to answer (%d request(s) given up on): holding back polls; "
"one canary getinfo waits for it\n", rpc_busy_gate_->abandoned());
canary_state_ = kCanaryRunning;
canary_thread_ = std::thread([this, cfg]() {
rpc::RPCClient client; // deliberately ungated: this is the one request we allow
{
std::lock_guard<std::mutex> lk(canary_mutex_);
canary_client_ = &client;
}
int result = kCanaryGone;
if (!shutting_down_ && client.open(cfg.host, cfg.port, cfg.rpcuser, cfg.rpcpassword, cfg.use_tls)) {
try {
rpc::RPCClient::TraceScope trace("Busy node / Canary");
client.call("getinfo", json::array(), 0L); // 0 = no client timeout
result = kCanaryAnswered;
} catch (const std::exception& e) {
const rpc::RpcFailureKind kind = rpc::rpcFailureKindOf(e);
if (kind == rpc::RpcFailureKind::AppError || kind == rpc::RpcFailureKind::Warmup)
result = kCanaryAnswered; // a worker ran it: cs_main is free again
else if (kind == rpc::RpcFailureKind::QueueFull)
result = kCanaryRefused;
}
}
{
std::lock_guard<std::mutex> lk(canary_mutex_);
canary_client_ = nullptr;
}
canary_state_ = result;
});
}
void App::tryConnect()
{
// Lite builds have no full node / RPC daemon, so never run the RPC connection state machine
@@ -324,8 +397,7 @@ void App::tryConnect()
if (++daemon_wait_attempts_ == kDaemonWaitWarnAttempts) {
ui::Notifications::instance().warning(TR("daemon_port_busy_warn"), 20.0f);
}
network_refresh_.setTimer(services::NetworkRefreshService::Timer::Core,
services::RefreshScheduler::kCoreDefault - 1.0f);
network_refresh_.deferFor(services::NetworkRefreshService::Timer::Core, 4.0f);
return;
}
daemon_wait_attempts_ = 0; // port is free — clear the bounded-wait counter
@@ -338,8 +410,7 @@ void App::tryConnect()
if (startEmbeddedDaemon()) {
// Will retry connection after daemon starts
VERBOSE_LOGF("[connect #%d] Embedded daemon starting, will retry connection...\n", connect_attempt);
network_refresh_.setTimer(services::NetworkRefreshService::Timer::Core,
services::RefreshScheduler::kCoreDefault - 1.0f);
network_refresh_.deferFor(services::NetworkRefreshService::Timer::Core, 1.0f);
} else {
// The daemon couldn't be started (binary not found, Sapling params missing, spawn
// failure, …). Surface the actual reason instead of leaving the status stuck on
@@ -356,8 +427,7 @@ void App::tryConnect()
detail.empty() ? std::string(TR("sb_daemon_start_failed")) : detail, 30.0f);
}
// Keep retrying: a missing binary/params can be fixed without a restart.
network_refresh_.setTimer(services::NetworkRefreshService::Timer::Core,
services::RefreshScheduler::kCoreDefault - 1.0f);
network_refresh_.deferFor(services::NetworkRefreshService::Timer::Core, 5.0f);
}
} else if (!isUsingEmbeddedDaemon()) {
VERBOSE_LOGF("[connect #%d] Embedded daemon disabled (using external). No config found at %s\n",
@@ -451,6 +521,7 @@ void App::tryConnect()
return [this, config, connected, warmingUp, warmupStatus, daemonStarting, externalDetected, attempt, connectErr,
connectKind]() {
noteDaemonRpcResult(connected, connectKind); // a busy node (timeouts / queue full) feeds the quit guard
if (connected) connect_backoff_.reset();
if (connected) {
VERBOSE_LOGF("[connect #%d] Connected successfully%s\n", attempt,
warmingUp ? " (daemon warming up)" : "");
@@ -525,10 +596,10 @@ void App::tryConnect()
// daemon's own console output) instead of a bare technical error.
const bool reachableButBusy = rpc::isRpcTimeout(connectKind);
applyDaemonInitStatus(reachableButBusy);
VERBOSE_LOGF("[connect #%d] RPC connection failed (%s) — daemon still starting, will retry...\n",
attempt, connectErr.c_str());
network_refresh_.setTimer(services::NetworkRefreshService::Timer::Core,
services::RefreshScheduler::kCoreDefault - 1.0f);
const float retryIn = connect_backoff_.next(connectKind);
VERBOSE_LOGF("[connect #%d] RPC connection failed (%s) — daemon still starting, retry in %.0fs\n",
attempt, connectErr.c_str(), retryIn);
network_refresh_.deferFor(services::NetworkRefreshService::Timer::Core, retryIn);
} else if (externalDetected) {
state_.connected = false;
// An external daemon is on the RPC port but not answering. A timeout means it's
@@ -542,10 +613,10 @@ void App::tryConnect()
} else {
connection_status_ = TR("sb_connecting_external");
}
VERBOSE_LOGF("[connect #%d] External daemon detected but RPC failed (%s), will retry...\n",
attempt, connectErr.c_str());
network_refresh_.setTimer(services::NetworkRefreshService::Timer::Core,
services::RefreshScheduler::kCoreDefault - 1.0f);
const float retryIn = connect_backoff_.next(connectKind);
VERBOSE_LOGF("[connect #%d] External daemon detected but RPC failed (%s), retry in %.0fs\n",
attempt, connectErr.c_str(), retryIn);
network_refresh_.deferFor(services::NetworkRefreshService::Timer::Core, retryIn);
} else {
onDisconnected("Connection failed");
VERBOSE_LOGF("[connect #%d] RPC connection failed — no daemon starting, no external detected\n", attempt);
@@ -692,6 +763,7 @@ void App::onConnected()
// behind the main refresh batch.
if (!fast_rpc_) {
fast_rpc_ = std::make_unique<rpc::RPCClient>();
fast_rpc_->setBusyGate(rpc_busy_gate_);
}
if (!fast_worker_) {
fast_worker_ = std::make_unique<rpc::RPCWorker>();
@@ -1800,7 +1872,11 @@ void App::refreshCoreData()
constexpr int kCoreFailuresBeforeDisconnect = 3;
if (!state_.warming_up) {
if (!result.balanceOk && !result.blockchainOk) {
if (++consecutive_core_failures_ >= kCoreFailuresBeforeDisconnect &&
// Only failures that put the connection itself in doubt count. A node that is merely
// busy (reply timeouts, a full queue, calls held back) is still there: tearing down
// then wiped the wallet state and restarted the connect probes that fill its queue.
if (rpc::rpcKindMeansLost(result.blockchainErrKind) &&
++consecutive_core_failures_ >= kCoreFailuresBeforeDisconnect &&
state_.connected) {
consecutive_core_failures_ = 0;
handleLostConnection("Lost connection to daemon");