feat(rpc): stop flooding a busy node's RPC queue; fix the reconnect storm

dragonxd serves RPC with 8 workers and a 16-deep queue. While a witness
rebuild holds cs_main, every request needing it waits, and a request the
client gave up on keeps its slot. The GUI kept polling and re-probing
until all 24 slots were its own abandoned requests (even stop was then
refused), treated the busy node as lost (tearing down the wallet state),
and retried connect at frame rate on some pages.

Busy gate (src/rpc/rpc_busy_gate.h), enforced in RPCClient::performCall
for the shared rpc_ / fast_rpc_:
- the first reply timeout or queue-full refusal marks the node busy;
  from then on nothing needing cs_main is sent (calls fail at once with
  the new RpcFailureKind::Deferred). stop, help and getrescaninfo are
  lock-free in dragonxd and always go through
- one canary getinfo with no client timeout waits on its own client and
  thread; its answer clears the gate and triggers an immediate refresh
  (queue-full: retried after 2..30 s; no answer: back to normal handling)
- the timer dispatch skips Core/Tx/Addresses/Peers/chat/mining/opid and
  connect attempts while busy
- the 3-strike teardown counts only failures that put the connection in
  doubt, never busy ones; no balance scans follow a busy chain-info call

Reconnect storm: retries were armed with setTimer(Core, kCoreDefault - 1),
i.e. "4 s elapsed", which against Overview's 2 s interval meant every
frame. RefreshScheduler::deferFor() is an absolute hold, and
ConnectRetryBackoff picks the delay by failure kind (not listening
1/2/5 s, queue full 2..30 s, busy 30 s, else 4 s).

Measured against the fake daemon: abandoned requests during a rebuild
went from ~24 (every slot) to at most 2, with no teardown and recovery
when the rebuild ends; a saturated queue sees about one canary per
back-off step instead of ~190 requests/s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-02 09:49:55 -05:00
co-authored by Claude Opus 5.5
parent 0c821333c8
commit 0f25c67600
12 changed files with 501 additions and 24 deletions
+15 -5
View File
@@ -499,6 +499,7 @@ bool App::init()
// Initialize RPC client
rpc_ = std::make_unique<rpc::RPCClient>();
rpc_->setBusyGate(rpc_busy_gate_);
// Initialize background RPC worker thread
worker_ = std::make_unique<rpc::RPCWorker>();
@@ -940,6 +941,10 @@ void App::update()
// enable the wallet UI), but there is no RPC daemon — so RPC polls (mining/balance/peers/txs)
// must key off rpc_ being genuinely connected or they'd fire and fail ("Not connected").
const bool rpcConnected = rpc_ && rpc_->isConnected();
// The node is too busy to answer: send it nothing that needs cs_main (it would only queue behind the
// rest and hold one of its few RPC slots) until the canary gets through. pumpRpcBusyCanary() runs it.
pumpRpcBusyCanary();
const bool rpcBusy = rpc_busy_gate_->busy();
// Fast refresh (mining stats + daemon memory) every second
// Skip when wallet is locked — no need to poll, and queued tasks
@@ -958,7 +963,7 @@ void App::update()
// Skip the mining poll while the daemon is in warmup (e.g. during -rescan). Otherwise
// getmininginfo is rejected with -28 ("Rescanning...") every second and floods the log.
// The rescan progress poll below still runs — that's how we track the warmup/rescan.
if (rpcConnected && !state_.warming_up && !runtime_rescan_active_) {
if (rpcConnected && !state_.warming_up && !runtime_rescan_active_ && !rpcBusy) {
refreshMiningInfo();
}
@@ -1210,7 +1215,7 @@ void App::update()
rpc::RPCWorker* opidWorker = (fast_worker_ && fast_worker_->isRunning())
? fast_worker_.get() : worker_.get();
if (network_refresh_.isDue(RefreshTimer::Opid) && !pending_opids_.empty()
&& rpcConnected && opidWorker && !opid_poll_in_progress_) {
&& rpcConnected && !rpcBusy && opidWorker && !opid_poll_in_progress_) {
network_refresh_.reset(RefreshTimer::Opid);
auto opids = pending_opids_; // copy for worker thread
opid_poll_in_progress_ = true;
@@ -1319,8 +1324,10 @@ void App::update()
}
// Per-category refresh with tab-aware intervals
// Skip when wallet is locked — same reason as above.
if (rpcConnected && !state_.isLocked()) {
// Skip when wallet is locked — same reason as above. Hold everything while the node is busy.
if (rpcConnected && !state_.isLocked() && rpcBusy) {
// nothing: the canary brings polling back (markImmediateRefresh) once the node answers
} else if (rpcConnected && !state_.isLocked()) {
const bool walletDataPage = currentPageNeedsWalletDataRefresh();
if (!runtime_rescan_active_ && network_refresh_.consumeDue(RefreshTimer::Core)) {
refreshCoreData();
@@ -1369,7 +1376,7 @@ void App::update()
if (network_refresh_.consumeDue(RefreshTimer::Peers)) {
refreshPeerInfo();
}
} else if (network_refresh_.consumeDue(RefreshTimer::Core)) {
} else if (!rpcBusy && network_refresh_.consumeDue(RefreshTimer::Core)) {
if (!connection_in_progress_ &&
wizard_phase_ == WizardPhase::None &&
!bootstrap_downloading_) {
@@ -5968,6 +5975,7 @@ void App::beginShutdown()
// the render loop exits, so the UI stays responsive.
if (rpc_) rpc_->requestAbort();
if (fast_rpc_) fast_rpc_->requestAbort();
abortRpcBusyCanary();
if (worker_) {
worker_->requestStop();
}
@@ -6874,6 +6882,8 @@ void App::shutdown()
// Ensure the mining-control thread is stopped + joined (idempotent; beginShutdown already did it on the
// normal quit path, but shutdown() can also run without it). Must precede xmrig_manager_ teardown.
stopMiningControlThread();
abortRpcBusyCanary();
if (canary_thread_.joinable()) canary_thread_.join();
// Wipe any copied secret from the OS clipboard before we exit — the 45s auto-clear timer
// never fires if the user quits sooner, which would otherwise leave a key/seed resident.