[POC] guix: produce a fully -static-pie bitcoind

[fanquake]

Use Guix to produce a fully -static-pie x86_64-linux-gnu bitcoind using GCC 13.2.0 and glibc 2.39. The binary is ~15mb stripped, includes wallet/upnp/natpmp/zmq etc.

# file bitcoind
	ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, for GNU/Linux 3.2.0, stripped

# ldd bitcoind
	statically linked

-static-pie

Produce a static position independent executable on targets that support it.
A static position independent executable is similar to a static executable,
but can be loaded at any address without a dynamic linker.

Using glibc for static builds, would be less of a leap compared to switching to musl libc (POC in #23203).

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.
A summary of reviews will appear here.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• #30022 (releases: use LLVM 18 for macOS by fanquake)
• #29987 (guix: build with glibc 2.31 by fanquake)
• #29881 (guix: use GCC 13 to builds releases by fanquake)
• #29790 ([DO NOT MERGE] cmake: Migrate CI scripts to CMake-based build system -- WIP by hebasto)
• #26950 (cleanse: switch to SecureZeroMemory for Windows cross-compile, check for usage by fanquake)
• #25391 (guix: Use LTO to build releases by fanquake)
• #24123 (guix: Pointer Authentication and Branch Target Identification for aarch64 Linux (Guix) by fanquake)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

[fanquake]

Rebased, and made small changes to fixup the linter and improve the security/symbol checks.

[theuni]

This is amazing!

Using glibc for static builds, would be less of a leap compared to switching to musl libc (POC in #23203).

Hard agree. Static musl builds would be very cool but tricky to audit for parity and thus pretty unlikely to ever become official/supported imo. This is a MUCH more palatable change.

[fanquake]

Added an additional patch to disable building various features/libs in GCC that we don't need. i.e libquadmath, libsanitizer, gcov, libada and native language support. This should speed up compiling GCC, and slightly reduces the size of the pthread patch. I'm not currently disabling these features via the available configure flags, because they get passed through to GMP, which cannot handle them.

I've also updated -flto usage to be -flto=auto, as that seems to now be the preferred invocation, and it mitigates GCC warning output. i.e:

lto-wrapper: warning: using serial compilation of 84 LTRANS jobs
lto-wrapper: note: see the '-flto' option documentation for more information

I've also split the ELF PIE check into it's own func, so we can special-case x86_64, and re-added the list of allowed ELF libs, but special-cased x86_64 to only pass if the list is empty.

[luke-jr]

How does this handle resolvers? eg, static glibc 2.33 applications break on systems with glibc 2.34...

[Frank-GER]

Which OS did you build on?

On Ubuntu 20.04 after build:
file bitcoind
bitcoind: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, for GNU/Linux 3.2.0, stripped

Trying to run it, it crashes after seconds:
addcon thread start
Loading addresses from DNS seed dnsseed.emzy.de.
bitcoind: dl-call-libc-early-init.c:37: _dl_call_libc_early_init: Assertion `sym != NULL' failed.
Aborted (core dumped)

Any ideas?

[fanquake]

Which OS did you build on?

The base OS shouldn't matter, because the build is performed with Guix. However I've been performing these builds on Ubuntu 22.04.

Any ideas?

How did you actually build bitcoind? Something like HOSTS="x86_64-linux-gnu" ./contrib/guix/guix-build?

[fanquake]

Rebased on master. Added a commit to use LIEF master, which reduces the number of changes in our scripts, as support for -static-pie ELF binaries has been added: lief-project/LIEF#747. Will switch to a new release when one is available.

[Frank-GER]

How did you actually build bitcoind? Something like HOSTS="x86_64-linux-gnu" ./contrib/guix/guix-build?

Yes, that's what I used to build it.
https://github.com/fanquake/bitcoin/commits/fully_static_bitcoind produces the exact same problem when building on Ubuntu 20.04

[fanquake]

https://github.com/fanquake/bitcoin/commits/fully_static_bitcoind produces the exact same problem when building on Ubuntu 20.04

I have just re-tested building the changes here (ef0edd5), and this produces a -static-pie binary as expected:

guix-build-ef0edd55f3bc/distsrc-ef0edd55f3bc-x86_64-linux-gnu/src/bitcoind: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, for GNU/Linux 3.2.0, with debug_info, not stripped

It's not clear to me that you are actually running a Guix build.

[Frank-GER]

the build I was testing:
~/bitcoin/guix-build-caa066465973/distsrc-caa066465973-x86_64-linux-gnu/src$ file bitcoind
bitcoind: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, for GNU/Linux 3.2.0, with debug_info, not stripped

just building again, fresh cloned with commit ef0edd5
export HOSTS='x86_64-linux-gnu'
./contrib/guix/guix-build

$ file guix-build-ef0edd55f3bc/distsrc-ef0edd55f3bc-x86_64-linux-gnu/src/bitcoind
guix-build-ef0edd55f3bc/distsrc-ef0edd55f3bc-x86_64-linux-gnu/src/bitcoind: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, for GNU/Linux 3.2.0, with debug_info, not stripped

$ ldd ./guix-build-ef0edd55f3bc/distsrc-ef0edd55f3bc-x86_64-linux-gnu/src/bitcoind
statically linked

$ ./bitcoind
...
2022-08-05T19:10:46Z init message: Starting network threads…
2022-08-05T19:10:46Z net thread start
2022-08-05T19:10:46Z dnsseed thread start
2022-08-05T19:10:46Z Loading addresses from DNS seed seed.bitcoinstats.com.
2022-08-05T19:10:46Z opencon thread start
2022-08-05T19:10:46Z msghand thread start
2022-08-05T19:10:46Z addcon thread start
2022-08-05T19:10:46Z init message: Done loading
bitcoind: dl-call-libc-early-init.c:37: _dl_call_libc_early_init: Assertion `sym != NULL' failed.
Aborted (core dumped)

build and run on Ubuntu 20.04
there is no other bitcoind on that OS

[Frank-GER]

building the same on Ubuntu 22.04:
file guix-build-ef0edd55f3bc/distsrc-ef0edd55f3bc-x86_64-linux-gnu/src/bitcoind
guix-build-ef0edd55f3bc/distsrc-ef0edd55f3bc-x86_64-linux-gnu/src/bitcoind: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), static-pie linked, for GNU/Linux 3.2.0, with debug_info, not stripped

ldd guix-build-ef0edd55f3bc/distsrc-ef0edd55f3bc-x86_64-linux-gnu/src/bitcoind
statically linked

Running it on Ubuntu22 it works.

Taking this bitcoind to Ubuntu 20 it shows:
file ./bitcoind
./bitcoind: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, for GNU/Linux 3.2.0, with debug_info, not stripped

and crashes with the above mentioned message:
bitcoind: dl-call-libc-early-init.c:37: _dl_call_libc_early_init: Assertion `sym != NULL' failed.
Aborted (core dumped)

Checksums are the same for both builds (on Ubuntu 20 and 22)
Ubuntu 20 sees it as dynamic linked, Ubuntu 22 as static-pie linked.
On Ubuntu 20 it crashes, on Ubuntu 22 it works.

Same bitcoind on Debian 10:
file ./bitcoind
./bitcoind: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, for GNU/Linux 3.2.0, with debug_info, not stripped
ldd ./bitcoind
statically linked
Gets past that point where it crashes on Ubuntu 20 and starts syncing

[fanquake]

Rebased. Simplified somewhat. Switched to GCC 13.2.0 and glibc 2.39.

[DrahtBot]

🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the
documentation.

Possibly this is due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.

Leave a comment here, if you need help tracking down a confusing failure.

_{Debug: https://github.com/bitcoin/bitcoin/runs/24492691529}