feat(rpc): stop flooding a busy node's RPC queue; fix the reconnect storm

dragonxd serves RPC with 8 workers and a 16-deep queue. While a witness
rebuild holds cs_main, every request needing it waits, and a request the
client gave up on keeps its slot. The GUI kept polling and re-probing
until all 24 slots were its own abandoned requests (even stop was then
refused), treated the busy node as lost (tearing down the wallet state),
and retried connect at frame rate on some pages.

Busy gate (src/rpc/rpc_busy_gate.h), enforced in RPCClient::performCall
for the shared rpc_ / fast_rpc_:
- the first reply timeout or queue-full refusal marks the node busy;
  from then on nothing needing cs_main is sent (calls fail at once with
  the new RpcFailureKind::Deferred). stop, help and getrescaninfo are
  lock-free in dragonxd and always go through
- one canary getinfo with no client timeout waits on its own client and
  thread; its answer clears the gate and triggers an immediate refresh
  (queue-full: retried after 2..30 s; no answer: back to normal handling)
- the timer dispatch skips Core/Tx/Addresses/Peers/chat/mining/opid and
  connect attempts while busy
- the 3-strike teardown counts only failures that put the connection in
  doubt, never busy ones; no balance scans follow a busy chain-info call

Reconnect storm: retries were armed with setTimer(Core, kCoreDefault - 1),
i.e. "4 s elapsed", which against Overview's 2 s interval meant every
frame. RefreshScheduler::deferFor() is an absolute hold, and
ConnectRetryBackoff picks the delay by failure kind (not listening
1/2/5 s, queue full 2..30 s, busy 30 s, else 4 s).

Measured against the fake daemon: abandoned requests during a rebuild
went from ~24 (every slot) to at most 2, with no teardown and recovery
when the rebuild ends; a saturated queue sees about one canary per
back-off step instead of ~190 requests/s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/rpc/rpc_busy_gate.h

@@ -0,0 +1,168 @@
+#pragma once
+
+// Keeping the GUI from flooding a busy node (S6). dragonxd's HTTP server has 8 workers and a 16-deep
+// queue. While a long operation holds cs_main (a witness rebuild can take an hour), every request that
+// needs cs_main waits, and a client that gives up does NOT free its slot. The GUI's polls used to time
+// out and be re-sent until all 24 slots were its own abandoned requests, after which even "stop" was
+// refused. The gate stops that: after the first sign the node is busy (a reply timeout, or a queue-full
+// refusal), the GUI's clients send nothing that needs cs_main until a single "canary" request (which
+// waits as long as it takes) gets through. Pure and header-only; shared by the GUI's RPC clients.
+
+#include "rpc_failure.h"
+
+#include <chrono>
+#include <mutex>
+#include <string>
+
+namespace dragonx {
+namespace rpc {
+
+// Kinds that mean "reached a node that is too busy to answer", as opposed to "no node there".
+inline bool rpcKindMeansBusy(RpcFailureKind kind)
+{
+    return kind == RpcFailureKind::ReplyTimeout || kind == RpcFailureKind::QueueFull ||
+           kind == RpcFailureKind::Deferred;
+}
+
+// Kinds that mean the connection itself is in doubt (counted toward a reconnect). Busy kinds, a
+// warmup reply, an application error and our own abort are not: the node is there.
+inline bool rpcKindMeansLost(RpcFailureKind kind)
+{
+    switch (kind) {
+    case RpcFailureKind::None:
+    case RpcFailureKind::ReplyTimeout:
+    case RpcFailureKind::QueueFull:
+    case RpcFailureKind::Deferred:
+    case RpcFailureKind::Warmup:
+    case RpcFailureKind::AppError:
+    case RpcFailureKind::Aborted:
+        return false;
+    default:
+        return true;
+    }
+}
+
+class RpcBusyGate {
+public:
+    // Methods dragonxd runs without cs_main: always sent, and their results say nothing about cs_main.
+    static bool isLockFree(const std::string& method)
+    {
+        return method == "stop" || method == "help" || method == "getrescaninfo";
+    }
+
+    // May this request be sent now? Lock-free methods always; the rest only while the node isn't busy.
+    bool admit(const std::string& method) const
+    {
+        if (isLockFree(method)) return true;
+        std::lock_guard<std::mutex> lk(mx_);
+        return !busy_;
+    }
+
+    // How a sent request went. Any real answer (even an error) means the node is serving cs_main
+    // requests again; a reply timeout or a queue-full refusal means it isn't. Other failures (nothing
+    // listening, auth, our own abort) say nothing about busyness and leave the state alone.
+    void record(const std::string& method, RpcFailureKind kind)
+    {
+        if (isLockFree(method)) return;
+        std::lock_guard<std::mutex> lk(mx_);
+        if (kind == RpcFailureKind::None || kind == RpcFailureKind::AppError || kind == RpcFailureKind::Warmup) {
+            clearLocked();
+        } else if (kind == RpcFailureKind::ReplyTimeout || kind == RpcFailureKind::QueueFull) {
+            if (!busy_) {
+                busy_ = true;
+                since_ = std::chrono::steady_clock::now();
+                ++busyPeriods_;
+            }
+            if (kind == RpcFailureKind::ReplyTimeout) ++abandoned_;   // it still holds a daemon slot
+        }
+    }
+
+    // The canary got through (or the node is gone): poll normally again.
+    void clear()
+    {
+        std::lock_guard<std::mutex> lk(mx_);
+        clearLocked();
+    }
+
+    bool busy() const
+    {
+        std::lock_guard<std::mutex> lk(mx_);
+        return busy_;
+    }
+    // Requests we gave up on that the node may still be holding, since the gate last cleared.
+    int abandoned() const
+    {
+        std::lock_guard<std::mutex> lk(mx_);
+        return abandoned_;
+    }
+    // Increments each time the gate turns busy: lets the owner arm exactly one canary per busy period.
+    int busyPeriods() const
+    {
+        std::lock_guard<std::mutex> lk(mx_);
+        return busyPeriods_;
+    }
+    double busySeconds() const
+    {
+        std::lock_guard<std::mutex> lk(mx_);
+        if (!busy_) return 0.0;
+        return std::chrono::duration<double>(std::chrono::steady_clock::now() - since_).count();
+    }
+
+private:
+    void clearLocked()
+    {
+        busy_ = false;
+        abandoned_ = 0;
+    }
+
+    mutable std::mutex mx_;
+    bool busy_ = false;
+    int abandoned_ = 0;
+    int busyPeriods_ = 0;
+    std::chrono::steady_clock::time_point since_{};
+};
+
+/**
+ * How long to wait before the next connection attempt after a failure of this kind. Each repeat of the
+ * same kind waits longer, up to its cap; a success or a different kind starts over.
+ *   nothing listening / connect timeout: 1, 2, 5s (the node is starting, or gone)
+ *   queue full: 2, 4, 8, 16, 30s (the queue is someone's; hammering it only adds to it)
+ *   busy (reply timeout / deferred): 30s; the canary brings the connection back as soon as it can
+ *   anything else: 4s
+ */
+class ConnectRetryBackoff {
+public:
+    float next(RpcFailureKind kind)
+    {
+        streak_ = (kind == last_) ? streak_ + 1 : 0;
+        last_ = kind;
+        switch (kind) {
+        case RpcFailureKind::NotListening:
+        case RpcFailureKind::ConnectTimeout: {
+            static const float steps[] = {1.0f, 2.0f, 5.0f};
+            return steps[streak_ < 2 ? streak_ : 2];
+        }
+        case RpcFailureKind::QueueFull: {
+            static const float steps[] = {2.0f, 4.0f, 8.0f, 16.0f, 30.0f};
+            return steps[streak_ < 4 ? streak_ : 4];
+        }
+        case RpcFailureKind::ReplyTimeout:
+        case RpcFailureKind::Deferred:
+            return 30.0f;
+        default:
+            return 4.0f;
+        }
+    }
+    void reset()
+    {
+        streak_ = 0;
+        last_ = RpcFailureKind::None;
+    }
+
+private:
+    RpcFailureKind last_ = RpcFailureKind::None;
+    int streak_ = 0;
+};
+
+}  // namespace rpc
+}  // namespace dragonx

src/rpc/rpc_client.cpp

@@ -360,6 +360,8 @@ json RPCClient::makePayload(const std::string& method, const json& params)
 
 std::string RPCClient::performCall(const std::string& method, const json& params, long& httpCodeOut)
 {
+    if (busy_gate_ && !busy_gate_->admit(method))
+        throw RpcTransportError(RpcFailureKind::Deferred, "RPC deferred: the node is busy", CURLE_OK, 0, false);
     emitRpcTrace(method);
 
     json payload = makePayload(method, params);
@@ -380,13 +382,20 @@ std::string RPCClient::performCall(const std::string& method, const json& params
         long requestSize = 0;
         curl_easy_getinfo(impl_->curl, CURLINFO_REQUEST_SIZE, &requestSize);
         const bool sent = requestSize > 0;
-        throw RpcTransportError(classifyRpcFailure(res, sent, 0, 0, std::string()),
-                                "RPC request failed: " + std::string(curl_easy_strerror(res)), res, 0, sent);
+        const RpcFailureKind kind = classifyRpcFailure(res, sent, 0, 0, std::string());
+        if (busy_gate_) busy_gate_->record(method, kind);
+        throw RpcTransportError(kind, "RPC request failed: " + std::string(curl_easy_strerror(res)), res, 0, sent);
     }
 
     // Check HTTP response code
     httpCodeOut = 0;
     curl_easy_getinfo(impl_->curl, CURLINFO_RESPONSE_CODE, &httpCodeOut);
+    if (busy_gate_) {
+        // Any HTTP answer means a daemon worker ran our request, except the queue-full refusal.
+        const bool queueFull = httpCodeOut == 500 &&
+                               response_data.find("Work queue depth exceeded") != std::string::npos;
+        busy_gate_->record(method, queueFull ? RpcFailureKind::QueueFull : RpcFailureKind::None);
+    }
 
     return response_data;
 }

src/rpc/rpc_client.h

@@ -6,6 +6,7 @@
 
 #include "types.h"
 #include "rpc_failure.h"   // RpcError, RpcTransportError, RpcFailureKind
+#include "rpc_busy_gate.h"
 #include <atomic>
 #include <string>
 #include <functional>
@@ -81,6 +82,14 @@ public:
      */
     RpcFailureKind trySendStop(long timeoutSec = 2);
 
+    /**
+     * @brief Share a busy gate with the app's other polling clients (nullptr: none)
+     *
+     * While the gate is busy, calls that need the daemon's cs_main fail at once with a Deferred
+     * RpcTransportError instead of being sent; every sent call reports how it went to the gate.
+     */
+    void setBusyGate(std::shared_ptr<RpcBusyGate> gate) { busy_gate_ = std::move(gate); }
+
     /**
      * @brief Disconnect from dragonxd
      */
@@ -297,6 +306,7 @@ private:
     std::string warmup_status_;
     std::string last_connect_error_;
     RpcFailureKind last_connect_kind_ = RpcFailureKind::None;
+    std::shared_ptr<RpcBusyGate> busy_gate_;
     std::string logged_connect_error_;      // last "Connection failed" text logged (rate limit)
     unsigned connect_error_repeats_ = 0;    // identical failures since it was logged
     json last_connect_info_;

src/rpc/rpc_failure.h

@@ -28,6 +28,7 @@ enum class RpcFailureKind {
     Warmup,          // JSON-RPC -28 (RPC_IN_WARMUP): the node is still loading
     AppError,        // the daemon ran the method and answered with a JSON-RPC error
     Aborted,         // our own requestAbort() cut the transfer
+    Deferred,        // not sent: the node is busy (RpcBusyGate), so we held back rather than queue more
     Other,
 };
 
@@ -45,6 +46,7 @@ inline const char* rpcFailureKindName(RpcFailureKind kind)
     case RpcFailureKind::Warmup:         return "Warmup";
     case RpcFailureKind::AppError:       return "AppError";
     case RpcFailureKind::Aborted:        return "Aborted";
+    case RpcFailureKind::Deferred:       return "Deferred";
     case RpcFailureKind::Other:          return "Other";
     }
     return "Other";