DanS
04ac7c1186
Binaries built on Ubuntu 22.04 require GLIBC_2.34 and GLIBCXX_3.4.30 and will not
start on Ubuntu 20.04 -- which is four of our five seeds, and an unknown share of
users. The binary the fleet actually runs today needs only GLIBC_2.29, so it was
built somewhere older; seed 176 has since been upgraded to 22.04 and now produces
binaries it is the only seed able to run.
--linux-compat and Dockerfile.compat already solved this (6d56ad854) but were
undocumented outside the build script and pinned to one base image. Parameterise
the base via ARG BASE_IMAGE (default unchanged, so --linux-compat behaves exactly
as before) and document the whole path.
doc/build-containers.md is written to be executed by a person or an agent starting
from a machine with nothing installed: why the glibc direction matters, with the
measured numbers; what already exists in the repo so nobody writes a second build
system; prerequisites and honest cost (~15GB, 4GB RAM, 1-2h per base because
depends/ builds boost, BDB, wolfssl and rust from source); one-target and
multi-target recipes; which base to pick and why 20.04 is the recommended floor
while 18.04 needs verifying (GCC 7 against -std=c++17); a mandatory verification
step with the exact objdump/readelf commands and the expected ceilings; and the
traps.
The traps are the part worth having written down: ETXTBSY when installing over a
running daemon (cp fails even after the process exits -- stage and rename, then
sha256-verify before starting); never touching configure.ac in a configured tree,
because the mtime alone triggers a reconfigure that dies on libdb_cxx; never
blind-touching a path that may not exist, which silently creates stray empty files;
RandomX needing ARCH=default or it emits AVX-512 that SIGILLs the fleet; build-win.sh
silently discarding every argument; and macOS being uncontainerisable because
depends/ has no darwin cross path at all.
Also records that full static linking is NOT the answer here: the daemon resolves
node1..node5.dragonx.is via getaddrinfo, and static glibc pushes that through NSS,
which dlopens libnss_dns at runtime and reintroduces the dependency it was meant to
remove.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
224 lines
10 KiB
Markdown
224 lines
10 KiB
Markdown
# Building release binaries in containers
|
||
|
||
Release binaries must be built in a container based on an **old** Linux distribution.
|
||
This document is written to be executed, by a person or an agent, on a machine that
|
||
has nothing set up yet.
|
||
|
||
---
|
||
|
||
## 1. Why this exists
|
||
|
||
glibc compatibility runs one way only. A binary linked against glibc 2.35 demands
|
||
symbol versions that glibc 2.31 does not have, and refuses to start. A binary linked
|
||
against glibc 2.29 runs on 2.29, 2.31 and 2.35 alike.
|
||
|
||
Measured on the actual fleet, 2026-08-25:
|
||
|
||
| binary | max GLIBC required | runs on |
|
||
|---|---|---|
|
||
| what all four 20.04 seeds run today (`v1.0.3-d159e7208`) | `GLIBC_2.29` | 18.04, 20.04, 22.04 |
|
||
| anything built on seed 176 today (Ubuntu 22.04) | `GLIBC_2.34` | 22.04 only |
|
||
|
||
The second binary will not start on four of our own five seeds. The loader reports:
|
||
|
||
```
|
||
/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
|
||
/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found
|
||
```
|
||
|
||
Nothing new is being *called*. glibc 2.34 merged libpthread and libdl into libc and
|
||
re-versioned every `pthread_*`, `dlsym` and `dladdr` symbol; 2.33 replaced the old
|
||
`__xstat` inlines with real `stat`/`fstat`/`lstat64`. All of those functions exist in
|
||
2.31 under older tags. Building against older headers is the entire fix.
|
||
|
||
**Do not try to solve this with full static linking.** The daemon calls `getaddrinfo`,
|
||
`gethostbyname` and `getnameinfo`, and it must resolve `node1..node5.dragonx.is`, which
|
||
are hard-coded and injected into `-addnode` on every node. Under a fully static glibc
|
||
binary those go through NSS, which `dlopen`s `libnss_dns.so.2` at run time — it either
|
||
fails or silently requires the target to have the same glibc you linked against, which
|
||
defeats the purpose.
|
||
|
||
---
|
||
|
||
## 2. What already exists in this repo
|
||
|
||
Do not write a new build system. Two pieces are already here:
|
||
|
||
- **`Dockerfile.compat`** — an Ubuntu base image that installs the toolchain, copies the
|
||
tree, **deletes any host-built `depends/` and object files**, runs `./util/build.sh`,
|
||
and strips the three binaries.
|
||
- **`./build.sh --linux-compat`** — builds that image, creates a throwaway container,
|
||
copies `dragonxd`, `dragonx-cli` and `dragonx-tx` out into
|
||
`release/dragonx-<version>-linux-amd64-ubuntu2004/`, adds `bootstrap-dragonx.sh`,
|
||
`asmap.dat` and the two sapling params, fixes ownership, and prints the binary's
|
||
maximum required GLIBC version.
|
||
|
||
The base image is parameterised via `ARG BASE_IMAGE` (default `ubuntu:20.04`), so the
|
||
same Dockerfile can target several glibc floors.
|
||
|
||
---
|
||
|
||
## 3. Prerequisites
|
||
|
||
Docker (the scripted path uses `docker` specifically; podman works for the manual path
|
||
if you alias or substitute it).
|
||
|
||
```sh
|
||
sudo apt-get update
|
||
sudo apt-get install -y docker.io git
|
||
sudo usermod -aG docker "$USER" # then log out and back in, or every command needs sudo
|
||
```
|
||
|
||
Budget, measured on a 4-core box:
|
||
|
||
| resource | needs |
|
||
|---|---|
|
||
| disk | ~15 GB free (the `depends/` tree alone is ~1.6 GB per target, plus image layers) |
|
||
| RAM | 4 GB minimum, 8 GB comfortable — the link step is the peak |
|
||
| time | **1–2 hours per base image on first build.** `depends/` builds boost, BDB, wolfssl, libevent, libsodium, libcurl and rust from source. Later builds reuse Docker layer cache unless the tree changed. |
|
||
|
||
`depends/` downloads and builds its own rust toolchain, so the host's rust (or absence
|
||
of it) is irrelevant.
|
||
|
||
---
|
||
|
||
## 4. Build one target
|
||
|
||
```sh
|
||
git clone https://git.dragonx.is/DragonX/dragonx
|
||
cd dragonx
|
||
git checkout <the tag or branch you are releasing>
|
||
|
||
./build.sh --linux-compat
|
||
```
|
||
|
||
Output lands in `release/dragonx-<version>-linux-amd64-ubuntu2004/` and the script
|
||
prints the max GLIBC at the end. `<version>` is read from `configure.ac`, not
|
||
hardcoded, so it always matches what the binaries report.
|
||
|
||
---
|
||
|
||
## 5. Build several targets
|
||
|
||
```sh
|
||
for BASE in ubuntu:18.04 ubuntu:20.04 ubuntu:22.04; do
|
||
TAG="dragonx-compat-${BASE#ubuntu:}"
|
||
TAG="${TAG//./}"
|
||
docker build --build-arg "BASE_IMAGE=$BASE" -f Dockerfile.compat -t "$TAG" .
|
||
|
||
OUT="release/dragonx-$(grep -oP 'define\(_CLIENT_VERSION_MAJOR, \K[0-9]+' configure.ac).$(grep -oP 'define\(_CLIENT_VERSION_MINOR, \K[0-9]+' configure.ac).$(grep -oP 'define\(_CLIENT_VERSION_REVISION, \K[0-9]+' configure.ac)-linux-amd64-${BASE#ubuntu:}"
|
||
mkdir -p "$OUT"
|
||
CID=$(docker create "$TAG")
|
||
for b in dragonxd dragonx-cli dragonx-tx; do docker cp "$CID:/build/src/$b" "$OUT/$b"; done
|
||
docker rm "$CID" >/dev/null
|
||
cp util/bootstrap-dragonx.sh contrib/asmap/asmap.dat sapling-output.params sapling-spend.params "$OUT/" 2>/dev/null || true
|
||
done
|
||
```
|
||
|
||
### Which base to choose
|
||
|
||
| base | glibc it provides | default GCC | verdict |
|
||
|---|---|---|---|
|
||
| `ubuntu:18.04` | 2.27 | 7 | **Verify before relying on it.** The tree is built with `-std=c++17`; GCC 7's C++17 support is incomplete and its cmake (3.10) may be too old for RandomX. Attempt only if you need to reach 18.04 users, and treat a successful build as the proof. |
|
||
| `ubuntu:20.04` | 2.31 | 9 | **Recommended floor.** GCC 9 covers C++17 fully. Evidence it works: the binary the fleet runs today requires only `GLIBC_2.29`, i.e. the code touches nothing newer, so a 20.04 build reaches 18.04 machines anyway. |
|
||
| `ubuntu:22.04` | 2.35 | 11 | **Do not ship this.** It is what we already have and what excludes four of our own seeds. Useful only for development. |
|
||
|
||
Ubuntu 20.04 left standard support in April 2025, which is precisely why it belongs in
|
||
a container on a patched host rather than on a build box someone has to maintain.
|
||
|
||
---
|
||
|
||
## 6. Verify — this step is not optional
|
||
|
||
A build that silently targets the wrong glibc looks completely normal until a user
|
||
reports that nothing starts.
|
||
|
||
```sh
|
||
BIN=release/dragonx-<version>-linux-amd64-ubuntu2004/dragonxd
|
||
|
||
# The ceiling. Must be <= the glibc of the OLDEST system you intend to support.
|
||
objdump -p "$BIN" | grep -oE 'GLIBC_2\.[0-9]+' | sort -t. -k2 -n | tail -1
|
||
objdump -p "$BIN" | grep -oE 'GLIBCXX_3\.4\.[0-9]+' | sort -t. -k3 -n | tail -1
|
||
|
||
# If the ceiling is too high, this names the symbols responsible.
|
||
readelf --dyn-syms --wide "$BIN" | grep -E '@GLIBC_2\.(3[2-9])'
|
||
```
|
||
|
||
Expected for a 20.04 build: `GLIBC_2.29` or lower, `GLIBCXX_3.4.26` or lower.
|
||
|
||
Then actually run it somewhere old. A ceiling check proves the loader will resolve the
|
||
symbols; it does not prove the binary works. `./dragonxd --version` on a real 20.04 box
|
||
is a ten-second confirmation.
|
||
|
||
---
|
||
|
||
## 7. Traps
|
||
|
||
Each of these has cost real time.
|
||
|
||
**`ETXTBSY` when installing over a running daemon.** `cp` onto the binary fails with
|
||
"Text file busy" *even after the process has exited* — `pgrep` returning nothing is not
|
||
sufficient, the kernel still holds the text mapping. Stage into the same directory and
|
||
`mv` (rename is not blocked), allow ~10 s to settle, and **sha256-verify the installed
|
||
file before starting it**. A failed copy that goes unnoticed leaves the old binary
|
||
running and looks like a successful deploy.
|
||
|
||
**Never touch `configure.ac` in a configured tree.** Even `cp`-ing back a byte-identical
|
||
copy updates its mtime, which makes `make` regenerate `aclocal.m4` and `configure` and
|
||
then re-run `configure`, which fails with `libdb_cxx headers missing` because the
|
||
depends prefix is not on the command line. If it happens: confirm
|
||
`git diff --quiet HEAD -- configure.ac`, then restore mtime order oldest-to-newest with
|
||
one-second gaps — `configure.ac`/`Makefile.am`, then `aclocal.m4`, then
|
||
`configure`/`Makefile.in`, then `config.status`, then `Makefile`. Inside a container
|
||
this cannot happen, which is one more reason to build there.
|
||
|
||
**Never blind-`touch` a path that might not exist.** `touch src/config/hush-config.h`
|
||
silently *creates* an empty stray file; the real header is `bitcoin-config.h`. Check
|
||
`git status` after any timestamp surgery.
|
||
|
||
**RandomX must be built with `ARCH=default`.** `util/build.sh` already passes it and the
|
||
comment there explains why: `ARCH=native` tunes to the build machine, and a build on an
|
||
AVX-512 host emitted 746 `zmm` instructions into `librandomx.a`, which `SIGILL`s on the
|
||
entire fleet. If you ever invoke cmake by hand, pass `-DARCH=default`.
|
||
|
||
**Strip before distributing.** Unstripped is ~220 MB, stripped ~16 MB. `Dockerfile.compat`
|
||
already strips inside the container.
|
||
|
||
**`util/build-win.sh` discards every argument.** There is no `"$@"` handling in it, so
|
||
`-j$(nproc)` and `--disable-tests` are dropped on the floor and the Windows build is
|
||
single-threaded. Expect it to be far slower than you planned.
|
||
|
||
**Windows also needs `-Wa,-mbig-obj` and `-DARCH=default`.** Both are in
|
||
`util/build-win.sh` today. The mingw flag was missing from `dev` for a month; without it
|
||
the cross-compile fails at link because boost-heavy translation units exceed the
|
||
PE/COFF section limit. Do not lose it on a re-branch.
|
||
|
||
**macOS cannot be containerised.** `util/build-mac.sh` is a native-Mac script, and there
|
||
is no darwin cross-compile path in `depends/` at all: `hosts/darwin.mk` wants
|
||
`native_cctools`, which has no package definition, and there is no SDK in the tree. It
|
||
also hardcodes an Intel Homebrew GCC path, so it produces x86_64 only — no arm64, no
|
||
universal binary. macOS needs a real Mac.
|
||
|
||
**The `contrib/gitian-descriptors/` files are not a build path.** They are unmodified
|
||
upstream Bitcoin files (`name: "bitcoin-win-0.11"`, suite `trusty`) with zero DragonX
|
||
content. Ignore them.
|
||
|
||
---
|
||
|
||
## 8. Handoff checklist
|
||
|
||
- [ ] Docker installed, user in the `docker` group, ~15 GB free
|
||
- [ ] Correct tag or branch checked out, tree clean (`git status`)
|
||
- [ ] Version in `configure.ac` is the one you intend to release
|
||
- [ ] `./build.sh --linux-compat` completes
|
||
- [ ] GLIBC ceiling is **2.31 or lower** (2.29 expected)
|
||
- [ ] GLIBCXX ceiling is **3.4.28 or lower** (3.4.26 expected)
|
||
- [ ] `dragonxd --version` runs on a real machine of the oldest supported distro
|
||
- [ ] Binaries stripped, `release/` contains the bootstrap script, `asmap.dat` and both sapling params
|
||
- [ ] sha256 recorded for each artifact
|
||
|
||
One more thing that is not a build step but belongs in the same conversation: the
|
||
in-app daemon updater refuses any release without a detached signature
|
||
(`kDaemonRequireSignature = true`). Publishing checksums alone means no existing user
|
||
can update in place.
|