/home/dev/lightwalletd/monitor_lwd.sh was carrying two uncommitted
production fixes, and the copy on disk had been reverted to the broken
committed version. The monitor that is actually running was executing a
deleted inode, so the fixes survived only as long as that process did --
any restart would have picked up the broken file.
The two fixes that were nearly lost:
* `-cache-size 5000` on the launch line. Without it a relaunch warms
the block cache from tip-400000 instead of tip-5000, which is
several minutes of getblock storm against the local node and several
minutes during which every wallet errors "Server's latest block is
behind ours".
* `EXIT_CODE=0; wait "$LWD_PID" || EXIT_CODE=$?` instead of
`wait "$LWD_PID" || true; EXIT_CODE=$?`. The latter reads the status
of `|| true` and is therefore always 0, so the monitor logged
"exited cleanly. Not restarting." and broke its loop on every exit
including crashes. That bug produced an 11h48m outage on 2026-08-21.
Recovered byte-identical from the running monitor via /proc/<pid>/fd/255
(md5 1823440d0af509c92583796af075b657) and committed so a checkout
cannot discard it again. An out-of-repo copy is kept at
/home/dev/monitor_lwd.sh.good.
Note the other branches still carry the broken blob; checking one out in
this working tree will clobber this file again. This working tree is a
live operational directory, not just a source checkout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FU87LdsJZiZkfq1eXubpeo
3.4 KiB
Executable File
3.4 KiB
Executable File