dragonx/qa/fd-guard/README.md

fd-guard: catch stray close() of files a thread doesn't own

This is the harness that found the wolfSSL CloseSocket macro bug. With that bug, P2P disconnects closed their fd number a second time, after another thread had reused it, and BDB then wrote wallet pages into blocks/*.dat.

It has two parts:

• shim/: an LD_PRELOAD guard around close, fclose, dup2 and dup3.

  • Before each close it resolves /proc/self/fd/N.
  • It logs a backtrace when a non-owner closes wallet.dat, database/log.* or blocks/*.dat, and for every close that fails with EBADF (the second half of a double close).
  • FDGUARD_ABORT=1 makes the first hit abort.

• repro/: isolated regtest runs that put a shimmed dragonxd under load:

  • rpc: impatient and RST clients, half-open connections, work-queue bursts and long-polls;
  • p2p: TLS peers churned by disconnectnode/ban/kill, plus misbehaving raw TCP/TLS clients;
  • tor: a fake control port;
  • mining: with blocknotify and autoshield.

  Each run executes in its own unprivileged network namespace: loopback only, no route.

Requirements

• Linux.
• unshare -rn (unprivileged user namespaces), nsenter, ip, python3, gcc, and addr2line for symbolizing.
• A dragonxd/dragonx-cli built with -g. Adding -rdynamic makes backtraces readable without addr2line.
• A regtest datadir with a grown wallet to copy from (GOLDEN). It is never modified.

Running

qa/fd-guard/shim/build.sh                      # builds libfdguard.so
qa/fd-guard/shim/run_tests.sh                  # self-test: double close, foreign close of a guarded file
cd qa/fd-guard/repro
export GOLDEN=/path/to/regtest/datadir BIN=/path/to/build/src   # RUNS defaults to ~/fd-guard-runs
LOADS=p2pmin,mine DISC_INT=5 GUARD=all ./go.sh p2pmin 300      # 1 peer, disconnectnode every 5 s
LOADS=rpc,p2p,tor,mine WQ=64 RT=8 GUARD=all ./go.sh full 1200  # everything, production-like work queue
python3 tools/tally.py "$RUNS/<name>/fdguard.log"              # who closed what
python3 ../shim/symbolize.py --summary "$RUNS/<name>/fdguard.log"
python3 tools/scan_bdb.py "$RUNS/<name>"/target/regtest/blocks/*.dat   # BDB pages inside block files

Each run writes run.txt (full log plus summary), fdguard.log and the target's debug.log and db.log under $RUNS/<name>/.

What to expect

Measured on a 2041-note regtest wallet, before and after the fix (net: stop wolfSSL's CloseSocket macro ...), under the same loads:

Run	Binary	EBADF closes	Non-owner closes (wallet/log/blk)	libevent Epoll failures	db.log
p2pmin, 300 s	before	51 (≈ 1 per disconnect)	1	0	empty
	after	0	0	0	empty
full, 1200 s, WQ=64 RT=8	before	930	186 (37)	111	632 KB: 7500 × BDB0060 PANIC, write failed for page N
	after	0	0	0	empty

• RPC-only and Tor-only loads give 0 reports, even before the fix: P2P disconnects are the only origin.
• scan_bdb.py found no BDB pages in block files in any run. The landing of a page in a block file needs the reused number to be a writable block file at the moment BDB writes. Production did hit that (blk00011/blk00020), but it is rare.
• SSL_write ... closing connection lines are expected under tcpchurn. They are how the load triggers disconnects, not a symptom.