/home/dev/lightwalletd is simultaneously a git checkout and the runtime
directory, and monitor_lwd.sh -- the primary's live supervisor -- was a
tracked file inside it. So a routine `git reset --hard` during an
unrelated cherry-pick reverted the running supervisor to an older
committed revision. That happened on 2026-08-26. Nothing noticed,
because the running monitor was executing an already-deleted inode: the
file on disk was broken while the live process was fine. It was
recovered from /proc/<pid>/fd/255.
What the reverted copy would have reintroduced, had it ever restarted:
the loss of `-cache-size 5000`, so every relaunch warms the block cache
from tip-400000 instead of tip-5000; and `wait "$LWD_PID" || true;
EXIT_CODE=$?`, which reads the exit status of `|| true` and is therefore
always 0, so the monitor logs "exited cleanly. Not restarting." and
breaks its loop on every exit including crashes -- the bug behind an
11h48m outage on 2026-08-21.
Move it to deploy/, from where it is copied to /home/dev/ and run. The
runtime directory now holds only the binary and its logs, so no
checkout, reset, rebase or branch switch can reach a running supervisor.
The script no longer derives its paths from its own location: SCRIPT_DIR
became an explicit LWD_DIR, because the script and the runtime directory
are deliberately no longer the same place. lwd_watchdog.sh launches
"$MONITOR" by absolute path and refuses to run if it is missing.
Verified: run from /tmp, the relocated monitor resolves its binary and
log through LWD_DIR rather than its own directory, and launches with
-cache-size 5000 intact. The running monitor was not restarted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
/home/dev/lightwalletd/monitor_lwd.sh was carrying two uncommitted
production fixes, and the copy on disk had been reverted to the broken
committed version. The monitor that is actually running was executing a
deleted inode, so the fixes survived only as long as that process did --
any restart would have picked up the broken file.
The two fixes that were nearly lost:
* `-cache-size 5000` on the launch line. Without it a relaunch warms
the block cache from tip-400000 instead of tip-5000, which is
several minutes of getblock storm against the local node and several
minutes during which every wallet errors "Server's latest block is
behind ours".
* `EXIT_CODE=0; wait "$LWD_PID" || EXIT_CODE=$?` instead of
`wait "$LWD_PID" || true; EXIT_CODE=$?`. The latter reads the status
of `|| true` and is therefore always 0, so the monitor logged
"exited cleanly. Not restarting." and broke its loop on every exit
including crashes. That bug produced an 11h48m outage on 2026-08-21.
Recovered byte-identical from the running monitor via /proc/<pid>/fd/255
(md5 1823440d0af509c92583796af075b657) and committed so a checkout
cannot discard it again. An out-of-repo copy is kept at
/home/dev/monitor_lwd.sh.good.
Note the other branches still carry the broken blob; checking one out in
this working tree will clobber this file again. This working tree is a
live operational directory, not just a source checkout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
Bump for the GetTransaction crash fix, and make the version one value
instead of two literals that could drift.
It was duplicated: cmd/server/main.go had `var version = "0.1.1"` for
--version, and frontend/service.go had "0.1.1-dragonxlightd" inline in
the LightdInfo reply. The gRPC one is the load-bearing copy -- it is
walletrpc/service.proto:48, so every client reads it, and it is the only
way to tell from off-box which build a node is running.
That property is the point of bumping now rather than later. With it, a
rollout can be verified by probing each endpoint over TLS and reading
the advertised version, instead of shelling in to compare binary
checksums, and instead of the only alternative positive test -- calling
GetTransaction on a mempool txid, which proves the fix by crashing any
node that does not have it.
Verified: --version prints 0.1.2, and a GetLightdInfo probe against a
test instance returns 0.1.2-dragonxlightd where production still returns
0.1.1-dragonxlightd.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
GetTransaction asserted the height out of getrawtransaction's reply
without checking it:
txHeight = txinfo.(map[string]interface{})["height"].(float64)
dragonxd emits "height" only for a transaction that is in a block --
rawtransaction.cpp puts it inside `if (!hashBlock.IsNull())` -- so every
mempool transaction comes back without the key. The assertion then runs
nil.(float64) and panics. Nothing recovers it: grpc-go v1.24.0 installs
no recovery interceptor (there is no recover() in its server.go) and
this daemon adds none, so the panic takes down the whole process and
every wallet connected to that endpoint with it.
Any client can trigger it deliberately: broadcast a transaction, then
ask for it before it is mined. GetMempoolStream, added in b1b0d45,
hands out unconfirmed txids by design, so ordinary 0-conf use walks
straight into it.
Verified against a live mempool transaction on this node, whose reply
has neither "height" nor "blockhash": the old expression panics with
"interface conversion: interface {} is nil, not float64"; the new one
returns cleanly.
An unconfirmed transaction is now reported as tip+1, which is what
GetMempoolStream already advertises for the same transactions
(mempool.go:109).
Also harden GetSaplingInfo, which had six more unchecked assertions on
the getblockchaininfo reply. Those run on the block-ingestor goroutine,
where a panic is equally fatal. The top-level object is asserted once
and every field is read with the comma-ok form; a missing field now
degrades instead of crashing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
