feat(rpc): stop flooding a busy node's RPC queue; fix the reconnect storm
dragonxd serves RPC with 8 workers and a 16-deep queue. While a witness rebuild holds cs_main, every request needing it waits, and a request the client gave up on keeps its slot. The GUI kept polling and re-probing until all 24 slots were its own abandoned requests (even stop was then refused), treated the busy node as lost (tearing down the wallet state), and retried connect at frame rate on some pages. Busy gate (src/rpc/rpc_busy_gate.h), enforced in RPCClient::performCall for the shared rpc_ / fast_rpc_: - the first reply timeout or queue-full refusal marks the node busy; from then on nothing needing cs_main is sent (calls fail at once with the new RpcFailureKind::Deferred). stop, help and getrescaninfo are lock-free in dragonxd and always go through - one canary getinfo with no client timeout waits on its own client and thread; its answer clears the gate and triggers an immediate refresh (queue-full: retried after 2..30 s; no answer: back to normal handling) - the timer dispatch skips Core/Tx/Addresses/Peers/chat/mining/opid and connect attempts while busy - the 3-strike teardown counts only failures that put the connection in doubt, never busy ones; no balance scans follow a busy chain-info call Reconnect storm: retries were armed with setTimer(Core, kCoreDefault - 1), i.e. "4 s elapsed", which against Overview's 2 s interval meant every frame. RefreshScheduler::deferFor() is an absolute hold, and ConnectRetryBackoff picks the delay by failure kind (not listening 1/2/5 s, queue full 2..30 s, busy 30 s, else 4 s). Measured against the fake daemon: abandoned requests during a rebuild went from ~24 (every slot) to at most 2, with no teardown and recovery when the rebuild ends; a saturated queue sees about one canary per back-off step instead of ~190 requests/s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -47,10 +47,16 @@ void RefreshScheduler::tick(float deltaSeconds)
|
||||
timers_.fast += delta;
|
||||
timers_.txAge += delta;
|
||||
timers_.opid += delta;
|
||||
for (float& h : holds_) {
|
||||
if (h > 0.0f) h = std::max(0.0f, h - delta);
|
||||
}
|
||||
}
|
||||
|
||||
bool RefreshScheduler::isDue(Timer timer) const
|
||||
{
|
||||
const float h = hold(timer);
|
||||
if (h > 0.0f) return false; // held by deferFor()
|
||||
if (h == 0.0f) return true; // the hold ran out: due now, whatever the interval
|
||||
float timerInterval = interval(timer);
|
||||
return timerInterval > 0.0f && timerRef(timer) >= timerInterval;
|
||||
}
|
||||
@@ -65,19 +71,27 @@ bool RefreshScheduler::consumeDue(Timer timer)
|
||||
void RefreshScheduler::reset(Timer timer)
|
||||
{
|
||||
timerRef(timer) = 0.0f;
|
||||
holdRef(timer) = -1.0f;
|
||||
}
|
||||
|
||||
void RefreshScheduler::markDue(Timer timer)
|
||||
{
|
||||
holdRef(timer) = -1.0f; // an explicit "now" (user action) overrides a backoff
|
||||
float timerInterval = interval(timer);
|
||||
timerRef(timer) = timerInterval > 0.0f ? timerInterval : 0.0f;
|
||||
}
|
||||
|
||||
void RefreshScheduler::setTimer(Timer timer, float seconds)
|
||||
{
|
||||
holdRef(timer) = -1.0f;
|
||||
timerRef(timer) = std::max(0.0f, seconds);
|
||||
}
|
||||
|
||||
void RefreshScheduler::deferFor(Timer timer, float seconds)
|
||||
{
|
||||
holdRef(timer) = std::max(0.0f, seconds);
|
||||
}
|
||||
|
||||
float RefreshScheduler::timer(Timer timer) const
|
||||
{
|
||||
return timerRef(timer);
|
||||
|
||||
Reference in New Issue
Block a user