test(daemon): add fake_dragonxd, a slow dragonxd stand-in for stop/kill testing
A scriptable fake dragonxd that reproduces, in minutes, the incident where the GUI hard-killed a node busy with an hour-long witness rebuild: - dragonxd's HTTP-RPC queue (8 workers + 16 queued, -rpcthreads / -rpcworkqueue honoured): a full queue answers an immediate plain-text 500 "Work queue depth exceeded", and a request whose client gave up keeps its slot until a worker reaches it - a global cs_main held by a "witness rebuild" that logs the real markers; every method except stop/help takes it - warmup -28 with the daemon's allow-list, an optional startup replay - dragonxd's shutdown order: stop (or SIGTERM on POSIX) unbinds RPC, 503s open connections, waits for the rebuild, then logs "Shutdown: In progress..." / "Shutdown: done" and exits 0 Everything is appended to <datadir>/fake_dragonxd.events; a missing "clean-exit" line means the process was killed. One portable C++17 file (POSIX + Winsock), OFF by default (-DDRAGONX_BUILD_FAKE_DAEMON=ON), built into <build>/fake_dragonxd/ so it can't shadow the real bin/dragonxd. run-gui-with-fake.sh runs the GUI against it in a throwaway HOME. Against master's GUI on Linux it reproduces the bug: closing during a rebuild queues stop behind cs_main-blocked workers, then the GUI sends SIGTERM and SIGKILL mid-rebuild, with no clean exit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -45,6 +45,7 @@ option(DRAGONX_ENABLE_EMBEDDED_DAEMON "Enable embedded dragonxd support" ON)
|
||||
option(DRAGONX_BUILD_LITE "Build ObsidianDragonLite variant without full-node features" OFF)
|
||||
option(DRAGONX_ENABLE_LITE_BACKEND "Enable real lite wallet backend integration" OFF)
|
||||
option(DRAGONX_ENABLE_CHAT "Enable the HushChat protocol/UI integration" ON)
|
||||
option(DRAGONX_BUILD_FAKE_DAEMON "Build the fake slow dragonxd test stub (tools/fake_dragonxd); never shipped" OFF)
|
||||
set(DRAGONX_LITE_BACKEND_LIBRARY "" CACHE FILEPATH "Path to a prebuilt SDXL-compatible lite backend library")
|
||||
set(DRAGONX_LITE_BACKEND_INCLUDE_DIR "" CACHE PATH "Optional include directory for SDXL-compatible lite backend headers")
|
||||
set(DRAGONX_LITE_BACKEND_EXTRA_LIBS "" CACHE STRING "Additional libraries needed by the SDXL-compatible lite backend")
|
||||
@@ -1105,6 +1106,27 @@ else()
|
||||
message(STATUS "wallet-rebuild helper: OFF (no Berkeley DB found; set DRAGONX_BDB_ROOT for release builds)")
|
||||
endif()
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# fake_dragonxd — a deliberately slow, scriptable dragonxd stand-in that reproduces the daemon-stop
|
||||
# incident (usage and the incident it models: tools/fake_dragonxd/README.md).
|
||||
# Test tooling only: OFF by default, never packaged, and written to <build>/fake_dragonxd/ so its
|
||||
# "dragonxd" name can't shadow the real bin/dragonxd.
|
||||
# -----------------------------------------------------------------------------
|
||||
if(DRAGONX_BUILD_FAKE_DAEMON)
|
||||
add_executable(fake_dragonxd tools/fake_dragonxd/fake_dragonxd.cpp)
|
||||
target_link_libraries(fake_dragonxd PRIVATE nlohmann_json::nlohmann_json)
|
||||
if(WIN32)
|
||||
target_link_libraries(fake_dragonxd PRIVATE ws2_32)
|
||||
else()
|
||||
find_package(Threads REQUIRED)
|
||||
target_link_libraries(fake_dragonxd PRIVATE Threads::Threads)
|
||||
endif()
|
||||
set_target_properties(fake_dragonxd PROPERTIES
|
||||
OUTPUT_NAME dragonxd
|
||||
RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/fake_dragonxd)
|
||||
message(STATUS "fake_dragonxd test stub: ON (${CMAKE_BINARY_DIR}/fake_dragonxd/)")
|
||||
endif()
|
||||
|
||||
# -----------------------------------------------------------------------------
|
||||
# Tests
|
||||
# -----------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
# fake_dragonxd
|
||||
|
||||
A deliberately slow, scriptable stand-in for `dragonxd`, used to reproduce and regression-test how
|
||||
ObsidianDragon stops (or kills) a busy node. It is test tooling: off by default, never packaged.
|
||||
|
||||
The incident it models (2026-09):
|
||||
1. A 13.6k-note wallet spent ~1 h in a witness rebuild that held `cs_main`.
|
||||
2. The GUI's own abandoned polls filled the node's 24 RPC slots, so `stop` was never delivered.
|
||||
3. On close, the GUI hard-killed the node mid-rebuild, so every restart replayed ~62k blocks and redid the rebuild.
|
||||
|
||||
What it copies from the real daemon (DragonX v1.3.0):
|
||||
|
||||
- **HTTP-RPC queue.** 8 workers plus a 16-deep queue (or `-rpcthreads` / `-rpcworkqueue` from the
|
||||
command line or `DRAGONX.conf`). When the queue is full the request gets an immediate plain-text
|
||||
`HTTP 500 "Work queue depth exceeded"`. A request whose client timed out **keeps its slot** until
|
||||
a worker reaches it.
|
||||
- **cs_main.** One global lock. Every method except `stop` and `help` takes it. A "witness rebuild"
|
||||
holds it for `FAKE_REBUILD_SECS` and logs the real markers:
|
||||
- `Reading blocks for witness rebuild: <off> / <total>` (Phase 1, abortable);
|
||||
- nothing during Phase 2, which ignores shutdown;
|
||||
- `BuildWitnessCache: rebuilt N note witness cache(s) to height H in Tms using 8 thread(s)` at the end.
|
||||
- **Warmup.** `-28` errors carrying the current init message, using the daemon's warmup allow-list.
|
||||
Optionally an `Activating best chain...` replay, which logs `UpdateTip` lines and stops between steps on shutdown.
|
||||
- **Shutdown.** Triggered by `stop`, or by SIGTERM/SIGINT on POSIX (Windows has no signal channel,
|
||||
as in v1.3.0). In order, the stub:
|
||||
1. unbinds the listener at once;
|
||||
2. answers new requests on open connections with `503`;
|
||||
3. waits for the rebuild to finish;
|
||||
4. logs `Shutdown: In progress...` and renames its main thread to `hush-shutoff` (Linux `comm`);
|
||||
5. answers never-run queued requests with `500 Unhandled request`;
|
||||
6. flushes for `FAKE_FLUSH_SECS`, logs `Shutdown: done`, and exits 0.
|
||||
- **Logging.** With `-printtoconsole`, untimestamped lines go to stdout (this is what the GUI uses on
|
||||
Linux and macOS). Otherwise, timestamped UTC lines go to `<datadir>/debug.log` (the Windows path the GUI tails).
|
||||
- **Datadir.** It takes `.lock`, writes `hushd.pid` (POSIX) and `.cookie` (when there is no
|
||||
`rpcpassword`), and uses the same default datadir as dragonxd (`~/.hush/DRAGONX`, `%APPDATA%\Hush\DRAGONX`).
|
||||
|
||||
Its replies are canned: `getinfo` reports version `1030050`, the balance is fixed, and there are two
|
||||
peers on TEST-NET addresses. Unknown methods get the real `-32601 Method not found` (HTTP 404).
|
||||
|
||||
## Events file
|
||||
|
||||
Everything that matters for a test is appended to `<datadir>/fake_dragonxd.events`, one
|
||||
`<name> t=<unix ms> [k=v ...]` per line:
|
||||
|
||||
| Event | Meaning |
|
||||
|---|---|
|
||||
| `start`, `listening`, `warmup-end` | startup; `start` lists every knob |
|
||||
| `rebuild-start`, `rebuild-end`, `rebuild-aborted` | the cs_main hold |
|
||||
| `queue-full` | the first `Work queue depth exceeded` |
|
||||
| `max-abandoned n=` | a new high for client-abandoned requests still holding a slot |
|
||||
| `stop-received n=` | a `stop` RPC ran (every one is logged, so resends are visible) |
|
||||
| `stop-rejected status=` | a `stop` never reached a worker (500 queue-full, or 503 after interrupt) |
|
||||
| `sigterm` | SIGTERM/SIGINT received (POSIX) |
|
||||
| `http-interrupted`, `shutdown-begin` | shutdown progress |
|
||||
| `stats ...` | counters, every 30 s when they change |
|
||||
| `clean-exit ...` | **graceful exit**, with final counters. If this is missing, the process was killed. |
|
||||
|
||||
## Build
|
||||
|
||||
```bash
|
||||
cmake -S . -B build/linux -DDRAGONX_BUILD_FAKE_DAEMON=ON
|
||||
cmake --build build/linux --target fake_dragonxd # -> build/linux/fake_dragonxd/dragonxd
|
||||
|
||||
# Windows (mingw cross, from Linux/WSL), with the toolchain ./build.sh --win-release writes; own tree
|
||||
cmake -S . -B build/windows-tests -DCMAKE_TOOLCHAIN_FILE=build/windows/mingw-toolchain.cmake \
|
||||
-DDRAGONX_USE_SYSTEM_SDL3=OFF -DDRAGONX_BUILD_FAKE_DAEMON=ON
|
||||
cmake --build build/windows-tests --target fake_dragonxd # -> build/windows-tests/fake_dragonxd/dragonxd.exe
|
||||
```
|
||||
|
||||
The output goes to its own `fake_dragonxd/` directory so it never replaces the real `bin/dragonxd`.
|
||||
|
||||
## Run the GUI against it (Linux)
|
||||
|
||||
```bash
|
||||
FAKE_REBUILD_SECS=300 tools/fake_dragonxd/run-gui-with-fake.sh [--scratch DIR]
|
||||
```
|
||||
|
||||
The script builds a throwaway `HOME` with:
|
||||
|
||||
- copies of the GUI and the stub (as `dragonxd`, next to the GUI, where `findDaemonBinary()` looks first);
|
||||
- a `DRAGONX.conf` and an empty `blocks/`, so the first-run wizard is skipped;
|
||||
- `settings.json` with `verbose_logging`.
|
||||
|
||||
It then starts the GUI and prints the events file when the GUI exits. It refuses to run if
|
||||
something already listens on 21769, because the GUI would adopt that process as an external node.
|
||||
To "close the window" from a script, send SIGTERM to the GUI (`kill -TERM <gui pid>`); SDL turns it
|
||||
into a normal quit.
|
||||
|
||||
To run the stub alone: `FAKE_... build/linux/fake_dragonxd/dragonxd -datadir=/some/dir [-printtoconsole]`
|
||||
with a `DRAGONX.conf` (`rpcuser`/`rpcpassword`/`rpcport`) in that dir.
|
||||
|
||||
## Windows
|
||||
|
||||
Use a **separate Windows user or a VM**:
|
||||
|
||||
- The datadir is `%APPDATA%\Hush\DRAGONX` and can't be redirected for the GUI.
|
||||
- Until S3 lands, the GUI `TerminateProcess`es **any** `dragonxd.exe` of the same user, including a real node.
|
||||
|
||||
Put `dragonxd.exe` (the stub) next to `ObsidianDragon.exe`. Set the `FAKE_*` variables in that
|
||||
user's environment, or start the stub by hand from a console for case E. Each new GUI-launched stub
|
||||
inherits the GUI's environment.
|
||||
|
||||
## Knobs (environment)
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `FAKE_WARMUP_SECS` | 3 | `-28 Loading block index...` phase |
|
||||
| `FAKE_REPLAY_SECS` / `FAKE_REPLAY_BPS` | 0 / 12 | startup replay (`Activating best chain...`, UpdateTip lines) |
|
||||
| `FAKE_REBUILD_DELAY_SECS` | 20 | wait after `Done Loading` before the rebuild, so the GUI connects first |
|
||||
| `FAKE_REBUILD_SECS` | 300 | how long the rebuild holds cs_main; `0` = healthy node, no rebuild |
|
||||
| `FAKE_PHASE1_SECS` | 2 | the logged, abortable part of the rebuild |
|
||||
| `FAKE_POST_REBUILD_LOCK_SECS` | 0 | cs_main held after the end line (models the 18–36 s flush + wallet `SetBestChain`) |
|
||||
| `FAKE_FLUSH_SECS` | 3 | `Shutdown: In progress...` → `Shutdown: done` |
|
||||
| `FAKE_UPDATETIP_SECS` | 36 | block interval once loaded |
|
||||
| `FAKE_NOISE_SECS` | 20 | an unrelated log line this often (defeats "newest line" heuristics) |
|
||||
| `FAKE_SERVER_TIMEOUT_SECS` | 30 | idle keep-alive connections are closed (`-rpcservertimeout`) |
|
||||
| `FAKE_PREFILL` | 0 | inject N abandoned requests when the rebuild starts (`24` = saturated) |
|
||||
| `FAKE_MINING` | 0 | internal miner on: `stop` waits for cs_main first (`GenerateBitcoins` join) |
|
||||
| `FAKE_STUCK_SHUTDOWN` | 0 | the Sep 29 orphan: hold the datadir lock, no listener, ignore SIGTERM, never exit |
|
||||
| `FAKE_HEIGHT`, `FAKE_REBUILD_TOTAL`, `FAKE_NOTES`, `FAKE_VERSION` | 3327145, 62583, 13632, 1030050 | cosmetic numbers |
|
||||
|
||||
## Test cases
|
||||
|
||||
"Today" is the behaviour of v2.0.1 / `master` @ `46189da`. "Fixed" is the target behaviour of the
|
||||
daemon-stop hardening work.
|
||||
|
||||
| Case | Setup | Today (bug) | Fixed |
|
||||
|---|---|---|---|
|
||||
| A | defaults; leave the GUI open ~5 min into the rebuild | ~24 abandoned, then queue-full 500s; overlay "Launching dragonxd…" | busy overlay; `max-abandoned` ≤ 7 |
|
||||
| B | close the GUI during the rebuild | Linux: `sigterm` at +20 s, killed at +30 s, no `clean-exit`. Windows: killed at +20 s, no `stop-received` | modal; `stop-received`/`sigterm` within 2 s; waits; `clean-exit` |
|
||||
| C | close so the rebuild ends ~8 s later | killed | `stop` retried, `clean-exit` |
|
||||
| D | `FAKE_PREFILL=24`, close | `stop` never delivered | POSIX via SIGTERM; Windows `stop` retried until it lands |
|
||||
| E | Windows: start the stub by hand with `FAKE_STUCK_SHUTDOWN=1`, launch the GUI, close it | "Found orphaned dragonxd.exe … forcing termination" | no kill |
|
||||
| F | Detach on the shutdown screen | n/a | GUI exits at once; stub runs on to `clean-exit` |
|
||||
| G | Overview page during the 500 phase; count `[connect #N]` per minute in the GUI log | ~1,800–3,200/min | ≤ ~2–4/min |
|
||||
File diff suppressed because it is too large
Load Diff
Executable
+86
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run ObsidianDragon against tools/fake_dragonxd in a throwaway, fully isolated HOME (Linux).
|
||||
#
|
||||
# tools/fake_dragonxd/run-gui-with-fake.sh [--build-dir DIR] [--scratch DIR] [-- GUI args...]
|
||||
#
|
||||
# FAKE_* variables in the environment pass through to the stub (see README.md). It never touches
|
||||
# the real ~/.hush or ~/.config: HOME, XDG_CONFIG_HOME and TMPDIR all point into the scratch dir.
|
||||
# Re-using a --scratch dir keeps its settings, conf and GUI log; the stub's events file is appended.
|
||||
set -euo pipefail
|
||||
|
||||
repo="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
build_dir="$repo/build/linux"
|
||||
scratch=""
|
||||
port=21769 # the GUI's external-node / port checks are hard-wired to the default RPC port
|
||||
|
||||
usage() { sed -n '2,8p' "$0" | sed 's/^# \{0,1\}//'; }
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--build-dir) build_dir="$(cd "$2" && pwd)"; shift 2 ;;
|
||||
--scratch) scratch="$2"; shift 2 ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
--) shift; break ;;
|
||||
*) echo "unknown option: $1" >&2; usage >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
gui="$build_dir/bin/ObsidianDragon"
|
||||
stub="$build_dir/fake_dragonxd/dragonxd"
|
||||
if [[ ! -x "$gui" ]]; then
|
||||
echo "missing $gui; build the app first (./build.sh)" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -x "$stub" ]]; then
|
||||
echo "missing $stub; build it with:" >&2
|
||||
echo " cmake -S . -B ${build_dir#"$repo"/} -DDRAGONX_BUILD_FAKE_DAEMON=ON && cmake --build ${build_dir#"$repo"/} --target fake_dragonxd" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ss -ltn 2>/dev/null | grep -qE "[:.]$port( |$)"; then
|
||||
echo "port $port is already in use (a real dragonxd?). The GUI would adopt it as an external node; stop it first." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
scratch="${scratch:-$(mktemp -d "${TMPDIR:-/tmp}/fake-dragonxd.XXXXXX")}"
|
||||
mkdir -p "$scratch"
|
||||
scratch="$(cd "$scratch" && pwd)"
|
||||
app="$scratch/app"
|
||||
home="$scratch/home"
|
||||
datadir="$home/.hush/DRAGONX"
|
||||
config_dir="$home/.config/ObsidianDragon"
|
||||
mkdir -p "$app" "$datadir/blocks" "$config_dir" "$scratch/tmp"
|
||||
|
||||
# Copies, not symlinks: the GUI looks for dragonxd next to /proc/self/exe, which resolves links.
|
||||
cp -f "$gui" "$app/ObsidianDragon"
|
||||
cp -f "$stub" "$app/dragonxd"
|
||||
ln -sfn "$build_dir/bin/res" "$app/res"
|
||||
for f in sapling-spend.params sapling-output.params asmap.dat; do
|
||||
if [[ -e "$build_dir/bin/$f" ]]; then ln -sf "$build_dir/bin/$f" "$app/$f"; fi
|
||||
done
|
||||
# The GUI also keeps a dragonxd copy in its daemon dir; keep that one the stub too.
|
||||
if [[ -e "$config_dir/dragonx/dragonxd" ]]; then cp -f "$stub" "$config_dir/dragonx/dragonxd"; fi
|
||||
|
||||
# A pre-made conf + blocks/ dir skips the first-run wizard (App::isFirstRun).
|
||||
if [[ ! -e "$datadir/DRAGONX.conf" ]]; then
|
||||
printf 'rpcuser=fake\nrpcpassword=fake-%s\nrpcport=%s\n' "$RANDOM$RANDOM" "$port" > "$datadir/DRAGONX.conf"
|
||||
fi
|
||||
if [[ ! -e "$config_dir/settings.json" ]]; then
|
||||
printf '{\n "wizard_completed": true,\n "verbose_logging": true\n}\n' > "$config_dir/settings.json"
|
||||
fi
|
||||
|
||||
cat <<EOF
|
||||
scratch : $scratch
|
||||
stub events : $datadir/fake_dragonxd.events
|
||||
GUI log : $config_dir/dragonx-debug.log
|
||||
FAKE_* : $(env | grep -E '^FAKE_' | tr '\n' ' ')
|
||||
EOF
|
||||
|
||||
rc=0
|
||||
HOME="$home" XDG_CONFIG_HOME="$home/.config" TMPDIR="$scratch/tmp" "$app/ObsidianDragon" "$@" || rc=$?
|
||||
|
||||
echo "--- GUI exited ($rc). Stub events:"
|
||||
cat "$datadir/fake_dragonxd.events" 2>/dev/null || echo "(none)"
|
||||
if pgrep -f "^$app/dragonxd" >/dev/null; then
|
||||
echo "--- the stub is still running (pid $(pgrep -f "^$app/dragonxd" | tr '\n' ' ')); it exits after its rebuild + flush."
|
||||
fi
|
||||
exit "$rc"
|
||||
Reference in New Issue
Block a user