perf(net): one balance scan per poll, filtered opstatus, a guarded merge scope

The Core poll ran z_gettotalbalance twice (minconf 0 and 1), each a whole-
wallet scan under dragonxd's cs_main. The spendable (minconf=1) call now
runs only after the display call succeeded, and only when something is
unconfirmed or pending or the exact total moved: about 95% of polls make
one scan instead of two. A forced post-send balance can no longer be lost
to a refused enqueue, and pending-send debits aren't re-applied each tick.

z_getoperationstatus is filtered to our opids with its real argument form
([[id,...]]; the old ["id"] was rejected), falling back to the unfiltered
call if a daemon refuses the filter. The Shield/Merge dialog's own polls
are filtered and session-guarded too, and finally reach Done; the dead
RPCClient wrappers are deleted.

The Merge/Consolidate scope scan becomes a tested state machine: one scan
at a time, failures latch behind a Retry button, only "node not ready"
retries by itself (10/30/60 s, at most 3), Lite never scans. Its 120 s
calls are maintenance calls: every main-thread disconnect aborts one in
flight first, so a lifecycle action no longer freezes the UI behind it.
An aborted merge that was already sent says it may still go through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/services/network_refresh_service.cpp

@@ -7,6 +7,7 @@
 #include <chrono>
 #include <cmath>
 #include <cstdlib>
+#include <limits>
 #include <map>
 #include <set>
 #include <unordered_map>
@@ -271,6 +272,9 @@ NetworkRefreshService::CoreRefreshResult NetworkRefreshService::parseCoreRefresh
         result.shieldedBalance = readBalanceString(totalBalance, "private");
         result.transparentBalance = readBalanceString(totalBalance, "transparent");
         result.totalBalance = readBalanceString(totalBalance, "total");
+        const auto total = totalBalance.find("total");
+        if (total != totalBalance.end() && total->is_string())
+            result.totalBalanceZat = parseMoneyZat(total->get<std::string>());
     }
     if (spendableBalance.is_object()) {   // confirmed totals (minconf=1); left unset on old daemons
         result.spendableShieldedBalance = readBalanceString(spendableBalance, "private");
@@ -290,7 +294,8 @@ NetworkRefreshService::CoreRefreshResult NetworkRefreshService::parseCoreRefresh
     return result;
 }
 
-NetworkRefreshService::CoreRefreshResult NetworkRefreshService::collectCoreRefreshResult(RefreshRpcGateway& rpc, bool includeBalance)
+NetworkRefreshService::CoreRefreshResult NetworkRefreshService::collectCoreRefreshResult(RefreshRpcGateway& rpc,
+                                                                                         const CoreRefreshPlan& plan)
 {
     json totalBalance;
     json spendableBalance;
@@ -328,7 +333,9 @@ NetworkRefreshService::CoreRefreshResult NetworkRefreshService::collectCoreRefre
 
     // Don't follow a chain-info call that found the node busy (a reply timeout or a full queue) with two
     // balance scans: they need the same cs_main, and each would only take another of its few RPC slots.
-    if (includeBalance && !behind && !rpc::rpcKindMeansBusy(blockErrKind)) {
+    bool balanceAttempted = false;
+    bool spendableQueried = false;
+    if (plan.includeBalance && !behind && !rpc::rpcKindMeansBusy(blockErrKind)) {
         // z_gettotalbalance is O(mapWallet) and holds the daemon's cs_main for its whole duration —
         // seconds on a large shielded wallet. Time it so the caller can throttle how often it polls
         // (balanceRefreshDue()), keeping balance scans from starving block connection.
@@ -340,9 +347,25 @@ NetworkRefreshService::CoreRefreshResult NetworkRefreshService::collectCoreRefre
             balanceErrKind = rpc::rpcFailureKindOf(e);
             DEBUG_LOGF("Balance error: %s\n", e.what());
         }
-        try {   // SPENDABLE total: minconf=1 (confirmed). If absent, spendable degrades to display in apply.
-            spendableBalance = rpc.call("z_gettotalbalance", json::array({1}));
-        } catch (...) {}
+        // A Deferred call was held back by the busy gate and never reached the daemon.
+        balanceAttempted = balanceErrKind != rpc::RpcFailureKind::Deferred;
+        // SPENDABLE total: minconf=1 (confirmed), a second whole-wallet scan. Only after the display call
+        // succeeded (its result is dropped otherwise), and only when it can differ from the display total:
+        // something is unconfirmed or pending (needSpendable), or the exact total moved since the last
+        // poll, which is how a new incoming 0-conf payment first shows. Skipped, spendable equals display
+        // in applyCoreRefreshResult and nothing reads as unconfirmed.
+        if (balanceOk) {
+            std::optional<std::int64_t> totalZat;
+            const auto total = totalBalance.is_object() ? totalBalance.find("total") : totalBalance.end();
+            if (total != totalBalance.end() && total->is_string()) totalZat = parseMoneyZat(total->get<std::string>());
+            const bool totalMoved = !totalZat || !plan.prevTotalZat || *totalZat != *plan.prevTotalZat;
+            if (plan.needSpendable || totalMoved) {
+                spendableQueried = true;
+                try {   // If it fails, spendable degrades to display in apply.
+                    spendableBalance = rpc.call("z_gettotalbalance", json::array({1}));
+                } catch (...) {}
+            }
+        }
         balanceScanMs = std::chrono::duration<double, std::milli>(
             std::chrono::steady_clock::now() - balanceStart).count();
     }
@@ -351,9 +374,65 @@ NetworkRefreshService::CoreRefreshResult NetworkRefreshService::collectCoreRefre
     result.balanceScanMs = balanceScanMs;
     result.blockchainErrKind = blockErrKind;
     result.balanceErrKind = balanceErrKind;
+    result.balanceAttempted = balanceAttempted;
+    result.spendableQueried = spendableQueried;
     return result;
 }
 
+std::optional<std::int64_t> NetworkRefreshService::parseMoneyZat(const std::string& text)
+{
+    constexpr std::int64_t kCoin = 100000000;
+    // Leaves room for up to 99999999 zatoshis of fraction on top of whole * 1e8.
+    constexpr std::int64_t kMaxWhole = std::numeric_limits<std::int64_t>::max() / kCoin - 1;
+    std::size_t i = 0;
+    const bool negative = !text.empty() && text[0] == '-';
+    if (negative) ++i;
+    std::int64_t whole = 0;
+    std::size_t wholeDigits = 0;
+    for (; i < text.size() && text[i] >= '0' && text[i] <= '9'; ++i, ++wholeDigits) {
+        const int digit = text[i] - '0';
+        if (whole > (kMaxWhole - digit) / 10) return std::nullopt;   // the amount would overflow
+        whole = whole * 10 + digit;
+    }
+    if (wholeDigits == 0) return std::nullopt;
+    std::int64_t fraction = 0;
+    if (i < text.size() && text[i] == '.') {
+        ++i;
+        std::size_t fractionDigits = 0;
+        for (; i < text.size() && text[i] >= '0' && text[i] <= '9'; ++i, ++fractionDigits) {
+            if (fractionDigits == 8) return std::nullopt;   // finer than a zatoshi: not a daemon amount
+            fraction = fraction * 10 + (text[i] - '0');
+        }
+        if (fractionDigits == 0) return std::nullopt;
+        for (; fractionDigits < 8; ++fractionDigits) fraction *= 10;
+    }
+    if (i != text.size()) return std::nullopt;
+    const std::int64_t zat = whole * kCoin + fraction;
+    return negative ? -zat : zat;
+}
+
+std::optional<std::int64_t> NetworkRefreshService::nextTotalBaseline(const CoreRefreshResult& result,
+                                                                    std::optional<std::int64_t> previous)
+{
+    if (!result.balanceOk) return previous;                                       // no fresh total
+    if (result.spendableQueried && !result.spendableTotalBalance) return std::nullopt;   // re-measure next poll
+    return result.totalBalanceZat;
+}
+
+bool NetworkRefreshService::spendableBalanceNeeded(const WalletState& state, bool pendingSendWork, bool forced)
+{
+    if (forced || pendingSendWork) return true;
+    if (state.last_balance_update == 0) return true;
+    if (state.unconfirmedBalance > 1e-9) return true;
+    // The address scan saw unconfirmed funds (z_listunspent/listunspent at minconf=0 with confirmations).
+    auto anyUnconfirmed = [](const std::vector<AddressInfo>& addresses) {
+        return std::any_of(addresses.begin(), addresses.end(), [](const AddressInfo& address) {
+            return address.balance - address.spendableBalance > 1e-9;
+        });
+    };
+    return anyUnconfirmed(state.z_addresses) || anyUnconfirmed(state.t_addresses);
+}
+
 NetworkRefreshService::MiningRefreshResult NetworkRefreshService::parseMiningRefreshResult(
     const json& miningInfo, bool miningOk, const json& localHashrate, bool hashrateOk, double daemonMemoryMb)
 {
@@ -1327,9 +1406,10 @@ NetworkRefreshService::OperationStatusPollResult NetworkRefreshService::parseOpe
     OperationStatusPollResult parsed;
     if (!result.is_array()) return parsed;
 
-    // We poll z_getoperationstatus with no filter (this daemon rejects the per-opid filtered form),
-    // so the result lists ALL operations. Only act on the ones we're tracking — otherwise an
-    // unrelated failed/old operation would fire a spurious error toast or pollute send state.
+    // The poll is filtered to our opids (buildOperationFilterParams), but falls back to the unfiltered
+    // form, which lists ALL operations, if a daemon rejects the filter. Only act on the ones we're
+    // tracking — otherwise an unrelated failed/old operation would fire a spurious error toast or
+    // pollute send state.
     const std::set<std::string> requested(requestedOpids.begin(), requestedOpids.end());
 
     std::set<std::string> reported;
@@ -1379,6 +1459,18 @@ NetworkRefreshService::OperationStatusPollResult NetworkRefreshService::parseOpe
     return parsed;
 }
 
+std::optional<json> NetworkRefreshService::buildOperationFilterParams(const std::vector<std::string>& opids)
+{
+    json ids = json::array();
+    for (const auto& opid : opids) {
+        if (!opid.empty()) ids.push_back(opid);
+    }
+    if (ids.empty()) return std::nullopt;
+    json params = json::array();
+    params.push_back(std::move(ids));
+    return params;
+}
+
 void NetworkRefreshService::applyConnectionInfoResult(WalletState& state, const ConnectionInfoResult& result)
 {
     if (!result.ok) return;

src/services/network_refresh_service.h

@@ -120,6 +120,20 @@ public:
         // node (ReplyTimeout / QueueFull) from a gone one (NotListening) without reading what().
         rpc::RpcFailureKind blockchainErrKind = rpc::RpcFailureKind::None;
         rpc::RpcFailureKind balanceErrKind = rpc::RpcFailureKind::None;
+        // The minconf=0 call reached the daemon (it may still have failed). False when the balance was
+        // skipped (behind, busy chain read) or held back by the busy gate (Deferred) before it was sent.
+        bool balanceAttempted = false;
+        bool spendableQueried = false;                 // the minconf=1 call was made
+        std::optional<std::int64_t> totalBalanceZat;   // the minconf=0 "total" in zatoshis, exact
+    };
+
+    // What one Core refresh fetches. The minconf=1 (spendable) z_gettotalbalance is a second whole-wallet
+    // scan under cs_main; it only tells something the minconf=0 call doesn't while funds are unconfirmed
+    // or pending, so it runs only when needSpendable or the exact total moved since prevTotalZat.
+    struct CoreRefreshPlan {
+        bool includeBalance = true;
+        bool needSpendable = true;
+        std::optional<std::int64_t> prevTotalZat;
     };
 
     struct MiningRefreshResult {
@@ -271,7 +285,24 @@ public:
     // includeBalance=false skips z_gettotalbalance (which takes the wallet lock + cs_main) and only
     // fetches getblockchaininfo — used while syncing, where the balance is incomplete anyway and the
     // wallet should minimise lock contention with block connection.
-    static CoreRefreshResult collectCoreRefreshResult(RefreshRpcGateway& rpc, bool includeBalance = true);
+    static CoreRefreshResult collectCoreRefreshResult(RefreshRpcGateway& rpc, const CoreRefreshPlan& plan);
+    static CoreRefreshResult collectCoreRefreshResult(RefreshRpcGateway& rpc, bool includeBalance = true)
+    {
+        CoreRefreshPlan plan;
+        plan.includeBalance = includeBalance;
+        return collectCoreRefreshResult(rpc, plan);
+    }
+    // An amount as the daemon formats it ("[-]digits[.up to 8 digits]") in zatoshis; nullopt for anything
+    // else or anything that would overflow. Exact, so "did the total move" never trips on rounding.
+    static std::optional<std::int64_t> parseMoneyZat(const std::string& text);
+    // Whether the next balance poll needs the minconf=1 (spendable) call whatever the total does: never
+    // fetched, unconfirmed or pending funds, or a forced refresh after the user's own send.
+    static bool spendableBalanceNeeded(const WalletState& state, bool pendingSendWork, bool forced);
+    // The total the next poll compares against (CoreRefreshPlan::prevTotalZat). None after a poll whose
+    // minconf=1 call was made but brought back no total (timeout, held back, error): that left spendable
+    // degraded to display, so the next poll must count the total as moved and measure it again.
+    static std::optional<std::int64_t> nextTotalBaseline(const CoreRefreshResult& result,
+                                                         std::optional<std::int64_t> previous);
     static MiningRefreshResult parseMiningRefreshResult(const nlohmann::json& miningInfo,
                                                         bool miningOk,
                                                         const nlohmann::json& localHashrate,
@@ -340,6 +371,11 @@ public:
         int pageSize = 100);
     static OperationStatusPollResult parseOperationStatusPoll(const nlohmann::json& result,
                                                               const std::vector<std::string>& requestedOpids);
+    // Params for z_getoperationstatus/result filtered to these opids: ONE argument, an array of ids, so
+    // [[id, ...]]. nullopt when no non-empty id remains: an empty filter means every op the node has
+    // retained. Built explicitly, because a one-element braced list of a json ({ids}) collapses to the
+    // inner array, which the daemon rejects ("JSON value is not an array as expected").
+    static std::optional<nlohmann::json> buildOperationFilterParams(const std::vector<std::string>& opids);
 
     struct DaemonRescanScan {
         bool  foundRescan = false;

src/services/note_consolidation.cpp

@@ -5,6 +5,7 @@
 #include "note_consolidation.h"
 
 #include <algorithm>
+#include <cctype>
 #include <map>
 
 namespace dragonx {
@@ -20,6 +21,7 @@ std::vector<NoteGroup> groupSpendableNotes(const nlohmann::json& zListUnspent)
             if (address.empty()) continue;
             if (n.value("confirmations", 0) < 1) continue;   // z_mergetoaddress spends confirmed notes only
             if (!n.value("spendable", true)) continue;       // watch-only: no key to spend it
+            if (noteIsLocked(n)) continue;                   // held by an operation in flight
             NoteGroup& g = byAddress[address];
             g.address = address;
             ++g.notes;
@@ -36,6 +38,13 @@ std::vector<NoteGroup> groupSpendableNotes(const nlohmann::json& zListUnspent)
     return out;
 }
 
+bool noteIsLocked(const nlohmann::json& zListUnspentEntry)
+{
+    if (!zListUnspentEntry.is_object()) return false;
+    const auto it = zListUnspentEntry.find("locked");
+    return it != zListUnspentEntry.end() && it->is_boolean() && it->get<bool>();
+}
+
 bool worthConsolidating(const NoteGroup& group, double fee)
 {
     // The daemon rejects a merge whose fee is more than what it nets (merged value < 2 x fee).
@@ -67,5 +76,107 @@ AddressRun consolidateInPlace(const std::string& address, const MergeCall& call,
     return run;
 }
 
+ScopeWant scopeScanWant(const ScopeScan& scan, const ScopeInputs& in)
+{
+    if (!in.mergeMode || scan.phase == ScopePhase::Loaded) return ScopeWant::Nothing;
+    if (scan.inFlight || in.busyWithOp) return ScopeWant::Nothing;
+    if (scan.phase == ScopePhase::Failed && !scan.autoRetry) return ScopeWant::Nothing;   // waits for Retry
+    if (in.now < scan.retryAt) return ScopeWant::WaitBackoff;
+    if (!in.nodeReady) return ScopeWant::WaitNode;
+    return ScopeWant::Post;
+}
+
+std::uint32_t scopeScanBegin(ScopeScan& scan, double now)
+{
+    scan.phase = ScopePhase::Loading;
+    scan.inFlight = true;
+    scan.inFlightGen = scan.gen;
+    scan.startedAt = now;
+    return scan.gen;
+}
+
+bool scopeScanFinish(ScopeScan& scan, std::uint32_t gen, const std::string& error, rpc::RpcFailureKind kind,
+                     double now)
+{
+    if (scan.inFlight && gen == scan.inFlightGen) scan.inFlight = false;
+    if (gen != scan.gen) return false;   // a scan from an earlier session: its data is not this one's
+    if (error.empty()) {
+        scan.phase = ScopePhase::Loaded;
+        scan.failures = 0;
+        scan.autoRetry = false;
+        scan.retryAt = 0.0;
+        scan.error.clear();
+        return true;
+    }
+    scan.phase = ScopePhase::Failed;
+    ++scan.failures;
+    scan.error = error;
+    scan.autoRetry = scopeFailureRetries(kind, error) && scan.failures <= kScopeMaxAutoRetries;
+    scan.retryAt = scan.autoRetry ? now + scopeRetryDelaySec(scan.failures) : 0.0;
+    return false;
+}
+
+void scopeScanOpen(ScopeScan& scan, double now)
+{
+    if (scan.inFlight && now - scan.startedAt > kScopeInFlightMaxSec) scan.inFlight = false;   // lost
+    if (scan.inFlight && scan.phase == ScopePhase::Loading) return;   // still loading for the last session
+    ++scan.gen;
+    scan.phase = ScopePhase::Idle;
+    scan.failures = 0;
+    scan.autoRetry = false;
+    scan.retryAt = 0.0;
+    scan.error.clear();
+}
+
+void scopeScanInvalidate(ScopeScan& scan)
+{
+    ++scan.gen;
+    scan.phase = ScopePhase::Idle;
+    scan.failures = 0;
+    scan.autoRetry = false;
+    scan.retryAt = 0.0;
+    scan.error.clear();
+}
+
+void scopeScanRetry(ScopeScan& scan)
+{
+    if (scan.phase != ScopePhase::Failed) return;
+    scan.phase = ScopePhase::Idle;
+    scan.failures = 0;
+    scan.autoRetry = false;
+    scan.retryAt = 0.0;
+    scan.error.clear();
+}
+
+double scopeRetryDelaySec(int failures)
+{
+    if (failures <= 1) return 10.0;
+    if (failures == 2) return 30.0;
+    return 60.0;
+}
+
+bool scopeFailureRetries(rpc::RpcFailureKind kind, const std::string& message)
+{
+    using K = rpc::RpcFailureKind;
+    switch (kind) {
+    case K::NotListening:
+    case K::ConnectTimeout:
+    case K::Warmup:
+    case K::Deferred:
+    case K::ShuttingDown:
+    case K::Unhandled:
+    case K::Aborted:
+        return true;
+    case K::Other: {
+        std::string lower = message;
+        std::transform(lower.begin(), lower.end(), lower.begin(),
+                       [](unsigned char c) { return static_cast<char>(std::tolower(c)); });
+        return lower.find("not connected") != std::string::npos;
+    }
+    default:
+        return false;   // ReplyTimeout, QueueFull, AppError, Auth, parse/type errors
+    }
+}
+
 }  // namespace services
 }  // namespace dragonx

src/services/note_consolidation.h

@@ -10,8 +10,12 @@
 // Shielded only: a transparent address can't receive a non-coinbase output on DragonX (private chain).
 // Pure apart from the merge call it is handed, so the test target can drive it.
 
+#include "rpc/rpc_failure.h"
+
 #include <nlohmann/json.hpp>
 
+#include <chrono>
+#include <cstdint>
 #include <functional>
 #include <string>
 #include <vector>
@@ -26,10 +30,14 @@ struct NoteGroup {
     double amount = 0.0;
 };
 
-// Spendable shielded notes grouped by address, most notes first. Transparent UTXOs and unconfirmed or
-// watch-only notes are left out (z_mergetoaddress only spends confirmed notes it has keys for).
+// Spendable shielded notes grouped by address, most notes first. Transparent UTXOs and unconfirmed,
+// watch-only or locked notes are left out (z_mergetoaddress only spends confirmed, unlocked notes it has
+// keys for).
 std::vector<NoteGroup> groupSpendableNotes(const nlohmann::json& zListUnspent);
 
+// A z_listunspent entry the wallet has locked, e.g. for an operation in flight ("locked": true).
+bool noteIsLocked(const nlohmann::json& zListUnspentEntry);
+
 // Worth consolidating in place: at least two notes, and more than one batch's fee to work with.
 bool worthConsolidating(const NoteGroup& group, double fee);
 
@@ -59,5 +67,59 @@ struct AddressRun {
 // locks a batch's notes before the call returns, so each call takes the next batch. At most maxBatches.
 AddressRun consolidateInPlace(const std::string& address, const MergeCall& call, int maxBatches = 20);
 
+// ── Scope scan (the Merge/Consolidate dialog's listunspent + z_listunspent) ─────────────────────────
+// Both calls are whole-wallet scans under the daemon's cs_main. The dialog used to post the pair again on
+// the frame after any failure, every frame for as long as it stayed open (and forever on Lite, which has
+// no node). This decides when a scan may be posted; the dialog owns one ScopeScan.
+
+inline double steadySeconds()   // ImGui::GetTime() stands still while the window is minimized
+{
+    return std::chrono::duration<double>(std::chrono::steady_clock::now().time_since_epoch()).count();
+}
+
+inline constexpr long   kScopeCallTimeoutSec = 120;   // each call; a large wallet can take minutes
+inline constexpr double kScopeInFlightMaxSec = 1800;  // a scan outstanding this long is presumed lost
+inline constexpr int    kScopeMaxAutoRetries = 3;
+
+enum class ScopePhase { Idle, Loading, Loaded, Failed };
+
+struct ScopeScan {
+    ScopePhase phase = ScopePhase::Idle;
+    std::uint32_t gen = 0;          // bumped per dialog session; a result from another session is dropped
+    bool inFlight = false;          // a scan is posted and its result not yet in (survives close/reopen)
+    std::uint32_t inFlightGen = 0;  // the session that posted it: only its result clears inFlight
+    double startedAt = 0.0;
+    int failures = 0;
+    bool autoRetry = false;         // the last failure was "node not ready yet": retry by itself
+    double retryAt = 0.0;
+    std::string error;
+};
+
+struct ScopeInputs {
+    bool mergeMode = false;
+    bool busyWithOp = false;   // a merge was submitted or is in flight
+    bool nodeReady = false;    // App::walletScanReady()
+    double now = 0.0;
+};
+
+enum class ScopeWant { Nothing, Post, WaitNode, WaitBackoff };
+
+ScopeWant scopeScanWant(const ScopeScan& scan, const ScopeInputs& in);
+// Mark a scan posted: Loading, in flight for this session. Returns the generation to finish it with.
+std::uint32_t scopeScanBegin(ScopeScan& scan, double now);
+// Record a scan's result. True when it belongs to the current session and succeeded: apply its data.
+bool scopeScanFinish(ScopeScan& scan, std::uint32_t gen, const std::string& error, rpc::RpcFailureKind kind,
+                     double now);
+// The dialog opened: start a new session, unless a scan of the last one is still loading (adopt it).
+void scopeScanOpen(ScopeScan& scan, double now);
+// The data went stale (reconnect): the next want is a fresh scan. An in-flight scan stays guarded.
+void scopeScanInvalidate(ScopeScan& scan);
+// The user pressed Retry after a failure.
+void scopeScanRetry(ScopeScan& scan);
+double scopeRetryDelaySec(int failures);   // 10, 30, then 60 s
+// Retry by itself only when the node wasn't ready (refused, warming up, held back as busy, aborted):
+// not after a reply timeout or a real error, each of which would cost another pair of full scans.
+bool scopeFailureRetries(rpc::RpcFailureKind kind, const std::string& message);
+
 }  // namespace services
 }  // namespace dragonx