DragonX/ObsidianDragon
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity

fix(tx): de-duplicate transparent rows across listtransactions pages

Browse Source 
Transparent history is gathered by skip-paging listtransactions. The
underlying list can shift between the per-page RPCs - a new
transaction arriving pushes one page's tail into the next page's head
- so a row could be collected twice and show up as a duplicate in
History.

- Pages are now de-duplicated on the daemon's stable per-output
  identity: txid, send/non-send class, and vout. Keying on the class
  rather than the exact category means a coinbase that matures
  (immature to generate) between two fetches is still recognised as
  the same row; keeping vout means two equal-value outputs of one
  transaction stay distinct. No real transaction row is dropped or
  duplicated.
- Rows with no txid or vout ("move") are never keyed and pass through
  (they are dropped downstream anyway), so the de-dup fails open.
- The raw page size still drives the stop condition and the next-run
  row hint, independent of how many rows were unique.

Tests cover a shifted page whose head repeats the prior page's last
row (dropped), a sibling output of the same tx (kept), and an
un-keyable row (kept).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
DanS
2026-10-14 07:21:07 -05:00
co-authored by Claude Opus 4.8
parent 6a6ac8cf99
commit 25ad23aee4
2 changed files with 55 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/services/network_refresh_service.cpp
+25 -2
View File
@@ -12,6 +12,7 @@
#include <map>
#include <set>
#include <unordered_map>
#include <unordered_set>
using json = nlohmann::json;
@@ -1172,11 +1173,33 @@ NetworkRefreshService::TransactionRefreshResult NetworkRefreshService::collectTr
// The first page holds last time's count plus headroom, so most wallets get every row in one
// call (re-paging 1000 rows at a time formats ~rows^2/2000 rows); skip-paging stays the fallback.
const int pageSize = transparentPageSize(snapshot.transparentRowHint);
std::unordered_set<std::string> seenPagedRows;
for (int skip = 0;; skip += pageSize) {
json page = rpc.call("listtransactions", json::array({"*", pageSize, skip}));
appendTransparentTransactions(result.transactions, knownTxids, page, snapshot.miningAddresses);
const std::size_t got = page.is_array() ? page.size() : 0;
result.transparentRowCount += got;
// C267: skip-paging walks a list that can shift between RPCs (a new tx pushes the
// previous page's tail into this page's head). De-dup on the daemon's stable per-output
// identity (txid, send/non-send class, vout). Key on the send/non-send CLASS, not the
// exact category, so a coinbase maturing (immature->generate) between the two fetches is
// still recognised as the same row; keep vout so two equal-value outputs of one tx stay
// distinct. Rows without txid or vout ("move") are never keyed and pass through
// (appendTransparentTransactions drops them anyway).
json uniq = json::array();
if (page.is_array()) {
for (const auto& row : page) {
auto txid = readOptional<std::string>(row, "txid");
auto vout = readOptional<int>(row, "vout");
if (txid && !txid->empty() && vout) {
const bool isSend =
readOptional<std::string>(row, "category").value_or("") == "send";
std::string key = *txid + '|' + (isSend ? 's' : 'r') + '|' + std::to_string(*vout);
if (!seenPagedRows.insert(std::move(key)).second) continue; // true duplicate
}
uniq.push_back(row);
}
}
appendTransparentTransactions(result.transactions, knownTxids, uniq, snapshot.miningAddresses);
result.transparentRowCount += got; // RAW page size: drives the stop test + next row hint
if (got < static_cast<std::size_t>(pageSize)) break;
}
result.transparentChainOk = true;
tests/test_phase4.cpp
+30
View File
@@ -2430,6 +2430,36 @@ void testNetworkRefreshRpcCollectors()
EXPECT_EQ(pagedResult.transactions.size(), static_cast<size_t>(1001));
EXPECT_EQ(pagedResult.transactions[0].txid, std::string("paged-1000"));
{
// C267: a tx added between page RPCs shifts the list, so the previous page's last row reappears at
// the head of the next page. Rows (txid, send/non-send, vout) already taken are dropped; a sibling
// output of the same tx (different vout) and an un-keyable row (no vout) are kept (fail-open).
Refresh::TransactionRefreshSnapshot dedupSnapshot;
MockRefreshRpc dedupRpc;
json page1 = json::array();
for (int i = 0; i < 1000; ++i) {
page1.push_back(json{{"txid", "dd-" + std::to_string(i)}, {"vout", 0}, {"category", "receive"},
{"amount", 0.01}, {"time", i}, {"confirmations", 10},
{"address", "R-dd"}});
}
dedupRpc.addResponse("listtransactions", page1);
dedupRpc.addResponse("listtransactions", json::array({
json{{"txid", "dd-999"}, {"vout", 0}, {"category", "receive"}, {"amount", 0.01},
{"time", 999}, {"confirmations", 10}, {"address", "R-dd"}}, // repeat: dropped
json{{"txid", "dd-999"}, {"vout", 1}, {"category", "receive"}, {"amount", 0.01},
{"time", 999}, {"confirmations", 10}, {"address", "R-dd"}}, // sibling vout: kept
json{{"txid", "dd-novout"}, {"category", "receive"}, {"amount", 0.02},
{"time", 1001}, {"confirmations", 9}, {"address", "R-dd"}} // no vout: kept
}));
auto dedupResult = Refresh::collectTransactionRefreshResult(dedupRpc, dedupSnapshot, 328, 0);
EXPECT_EQ(dedupRpc.calls.size(), static_cast<size_t>(2));
EXPECT_EQ(dedupResult.transactions.size(), static_cast<size_t>(1002));
EXPECT_EQ(dedupResult.transparentRowCount, static_cast<size_t>(1003)); // RAW count, not deduped
size_t dd999 = 0;
for (const auto& t : dedupResult.transactions) if (t.txid == "dd-999") ++dd999;
EXPECT_EQ(dd999, static_cast<size_t>(2)); // vout 0 once + vout 1
}
// The pass floor: an address scanned below it is scanned again; without a floor (no pass running)
// only never-scanned addresses are. A pass's floor is its start height, so it completes however far
// the tip moves meanwhile (this is what unsticks the perpetual "refreshing history" state).