qa: add fd-guard, the close() guard and regtest harness that found the double close

shim/ is an LD_PRELOAD guard around close/fclose/dup2/dup3:
- before each close it resolves /proc/self/fd/N;
- it reports, with a backtrace, a non-owner close of wallet.dat,
  database/log.* or blocks/*.dat, and every close that fails with EBADF
  (the second half of a double close);
- FDGUARD_ABORT=1 aborts on the first hit;
- run_tests.sh self-tests it (51 checks).

repro/ runs a shimmed dragonxd on isolated regtest. Each run gets its
own unprivileged network namespace (loopback only). The loads are:
- rpc: a hammer with impatient/RST/half-open clients, work-queue
  bursts and long-polls;
- p2p: TLS peers churned by disconnectnode, ban and kill, plus a raw
  TCP/TLS misbehaver;
- tor: a fake control port;
- mine: mining with blocknotify and autoshield.
tools/tally.py groups the reports by closer. tools/scan_bdb.py finds BDB
pages written into block files.

With the wolfSSL CloseSocket macro in effect:
- p2pmin (300 s) gives one EBADF double close per disconnect;
- the full load (1200 s) gives 930 EBADF closes, 186 non-owner closes
  (37 of wallet/log/blk files) and a 632 KB db.log with BDB0060 PANIC;
- RPC-only and Tor-only loads give none.
After the fix, every count is 0 and db.log stays empty. The README has
the commands and the numbers.

Paths come from the environment (GOLDEN, BIN, RUNS, SHIM, ASMAP and the
Sapling params). Run output goes to $RUNS, outside the tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-16 21:23:01 -05:00
co-authored by Claude Opus 5.5
parent 70b35ac11a
commit 7131b3673f
16 changed files with 3594 additions and 0 deletions
+55
View File
@@ -0,0 +1,55 @@
# fd-guard: catch stray close() of files a thread doesn't own
This is the harness that found the wolfSSL `CloseSocket` macro bug. With that bug, P2P disconnects closed their fd number a second time, after another thread had reused it, and BDB then wrote wallet pages into `blocks/*.dat`.
It has two parts:
- `shim/`: an `LD_PRELOAD` guard around `close`, `fclose`, `dup2` and `dup3`.
- Before each close it resolves `/proc/self/fd/N`.
- It logs a backtrace when a non-owner closes `wallet.dat`, `database/log.*` or `blocks/*.dat`, and for every close that fails with `EBADF` (the second half of a double close).
- `FDGUARD_ABORT=1` makes the first hit abort.
- `repro/`: isolated regtest runs that put a shimmed `dragonxd` under load:
- **rpc:** impatient and RST clients, half-open connections, work-queue bursts and long-polls;
- **p2p:** TLS peers churned by `disconnectnode`/ban/kill, plus misbehaving raw TCP/TLS clients;
- **tor:** a fake control port;
- **mining:** with `blocknotify` and autoshield.
Each run executes in its own unprivileged network namespace: loopback only, no route.
## Requirements
- Linux.
- `unshare -rn` (unprivileged user namespaces), `nsenter`, `ip`, python3, gcc, and `addr2line` for symbolizing.
- A `dragonxd`/`dragonx-cli` built with `-g`. Adding `-rdynamic` makes backtraces readable without `addr2line`.
- A regtest datadir with a grown wallet to copy from (`GOLDEN`). It is never modified.
## Running
```sh
qa/fd-guard/shim/build.sh # builds libfdguard.so
qa/fd-guard/shim/run_tests.sh # self-test: double close, foreign close of a guarded file
cd qa/fd-guard/repro
export GOLDEN=/path/to/regtest/datadir BIN=/path/to/build/src # RUNS defaults to ~/fd-guard-runs
LOADS=p2pmin,mine DISC_INT=5 GUARD=all ./go.sh p2pmin 300 # 1 peer, disconnectnode every 5 s
LOADS=rpc,p2p,tor,mine WQ=64 RT=8 GUARD=all ./go.sh full 1200 # everything, production-like work queue
python3 tools/tally.py "$RUNS/<name>/fdguard.log" # who closed what
python3 ../shim/symbolize.py --summary "$RUNS/<name>/fdguard.log"
python3 tools/scan_bdb.py "$RUNS/<name>"/target/regtest/blocks/*.dat # BDB pages inside block files
```
Each run writes `run.txt` (full log plus summary), `fdguard.log` and the target's `debug.log` and `db.log` under `$RUNS/<name>/`.
## What to expect
Measured on a 2041-note regtest wallet, before and after the fix (`net: stop wolfSSL's CloseSocket macro ...`), under the same loads:
| Run | Binary | EBADF closes | Non-owner closes (wallet/log/blk) | libevent `Epoll` failures | `db.log` |
|---|---|---|---|---|---|
| `p2pmin`, 300 s | before | 51 (≈ 1 per disconnect) | 1 | 0 | empty |
| | after | 0 | 0 | 0 | empty |
| `full`, 1200 s, `WQ=64 RT=8` | before | 930 | 186 (37) | 111 | 632 KB: 7500 × `BDB0060 PANIC`, `write failed for page N` |
| | after | 0 | 0 | 0 | empty |
- RPC-only and Tor-only loads give 0 reports, even before the fix: P2P disconnects are the only origin.
- `scan_bdb.py` found no BDB pages in block files in any run. The landing of a page in a block file needs the reused number to be a writable block file at the moment BDB writes. Production did hit that (blk00011/blk00020), but it is rare.
- `SSL_write ... closing connection` lines are expected under `tcpchurn`. They are how the load triggers disconnects, not a symptom.