figure out how to statically link clang itself #150
Comments
|
Regarding linking statically, I know that linking against libz.a has a CMake issue (https://lists.llvm.org/pipermail/llvm-dev/2021-May/150505.html). We may also want to use -DLLDB_ENABLE_LIBXML2=off and disable libpython to reduce some complexity (libxml2 and libpython are only used by lldb). |
|
When I looked into it briefly for Rust for Linux, with the same idea (uploading it to kernel.org), it was indeed clear it was either not possible, not supported, or if it was both, not clear how to do it. But I agree, having a static build up there would be great. |
|
I can give it a try at some point! |
|
@serge-sans-paille did I understand correctly in |
|
no, not fully as hinted by this thread. Let me give it a try. |
|
@nickdesaulniers I reached that point: |
|
I would love to have the recipe documented and supported by Clang, ideally with a configuration option... Perhaps I am asking too much :) |
|
For performance, a statically linked clang is no faster than a -fvisibility-inlines-hidden + -Bsymbolic-functions + -fno-semantic-interposition clang. We will soon switch to If statically linking requires some non-trivial changes to the llvm-project build system, I will object to that. |
That is not the only reason to have statically linked binaries.
It is clear users want it, though. |
|
@ojeda : simple enough: from your build dir (obviously you need to adapt your path) $ cmake3 ../llvm -DLLVM_ENABLE_PROJECTS=clang -DTERMINFO_LIB=/opt/notnfs/sergesanspaille/install/lib/libncurses.a -DZLIB_LIBRARY_RELEASE=/opt/notnfs/sergesanspaille/install/lib/libz.a
[...]
$ make -j50 clang
[...]
$ ldd ./bin/clang
linux-vdso.so.1 (0x00007fff4b3ca000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe53549d000)
librt.so.1 => /lib64/librt.so.1 (0x00007fe535492000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fe53548b000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fe535292000)
libm.so.6 => /lib64/libm.so.6 (0x00007fe53514c000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe535132000)
libc.so.6 => /lib64/libc.so.6 (0x00007fe534f67000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe5354f9000)@MaskRay so no change to the configuration, just some usage of hidden cmake variables :-) |
|
@serge-sans-paille Thanks! That is helpful. I meant officially, in any case, i.e. having static builds supported by LLVM :) |
|
@serge-sans-paille that's closer, but I'm looking for a binary of clang that produces no output from @MaskRay less so concerned with performance than being able to distribute a binary that works on various distributions without any concerns for depencencies (if possible). I wonder if llvm-bazel can produce such an image? Looking at EDIT: looks like the llvm-bazel scripts don't produce a statically linked clang binary. :( |
|
@nickdesaulniers I can't say for bazel, but the following works like a charm for me: |
I tried on the debian package of apt.llvm.org and haven't seen a significant diff with these 3 flags. Without these flags: With: Note that I might not clear all the right caches (please let me know if this is the case) |
I have added
Clang ships a resource directory with built-in headers and runtime libraries. So you cannot copy a clang to another machine and expect it to work. You need to at least provide the built-in headers. In that case, Shipping two additional .so files libLLVM.so and libclang-cpp.so doesn't add much inconvenience. |
|
Great idea. @nickdesaulniers I am interested in a self-contained toolchain as well. However, our angles are a bit different. I want to have a toolchain that can reliably (cross-)compile fully static C/C++ applications. I use Bazel (which specifies all dependencies) and the latest C++ language features in my C++ projects. It is a PITA to work with the toolchains provided by the OS, as they are often outdated. On some distros (e.g. Ubuntu), it is possible to fetch from LLVM repo, but that makes the workflows unportable, and the fetched toolchains still depend on libs/headers from the system. From my experience, glibc is not well suited for static linking. Alternatively, there might be a way to improve distro compatibility without static linking: Enterprise Linux provides |
|
I think there's an existing flag LLVM_BUILD_STATIC for this. |
|
nixpkgs can link clang statically as of NixOS/nixpkgs#152776 (currently this change is merged into the The build is set up in the following way:
If you want to get a sense for the other flags passed, here are the definitions of llvm, clang and lld. There's also a talk on LLVM in nixpkgs (which got split up, so you'll have to jump around a bit) if you are interested in a more high level overview. I'm not sure how generally (that is outside of nixpkgs) the resulting statically linked clang binaries are, I suppose it depends on how many Nix-specific non-FHS paths leak into the binaries and if they are relevant for their operation. Happy to answer any other questions you have on this here as well. |
|
I might add that we could ditch our usual wrapper scripts to build something from Nix that is more something more minimal and relocatable, and easy to use anywhere. |
This image still has object files built with GCC (from library dependencies). But it's a start to start rebuilding dependencies from scratch. Link: ClangBuiltLinux/tc-build#150
This image still has object files built with GCC (from library dependencies). But it's a start to start rebuilding dependencies from scratch. Link: ClangBuiltLinux/tc-build#150
|
Any progress on this? |
Some progress was made at https://github.com/ClangBuiltLinux/containers/tree/main/llvm-project but nothing that is fully distributable as far as I remember (it has been some time since we have been able to work on this). |
|
Have you checked ellcc.org? |
cc @rdpennington - yeah,
I've had some success cross-building static toolchains with |
|
After testing, static linking of clang can improve performance https://gist.github.com/lateautumn233/382e1fd6ab09a51396b4abbe1711b766 |
|
@lateautumn233 to do benchmarking, you should use tools like https://github.com/sharkdp/hyperfine/ |
|
Dynamic linking uqiu@quqiu-laptop dev]$ bash kcbench -d -j 16 -i 3
Processor: AMD Ryzen 7 5700U with Radeon Graphics [16 CPUs]
Cpufreq; Memory: powersave [amd-pstate-epp]; 15370 MiB
Linux running: 6.8.6-2-cachyos [x86_64]
Compiler: 阿菌•未霜 clang version 19.0.0git (https://github.com/llvm/llvm-project.git e32c4dfefcd1d54eb8f353f6fa08ef6f06d0fcc4)
Linux compiled: 6.8.0 [/home/quqiu/.cache/kcbench/linux-6.8/]
Config; Environment: defconfig; CCACHE_DISABLE="1" LLVM="/home/quqiu/dev/amd64-kernel-arm/bin/"
Build command: make vmlinux
Filling caches: This might take a while... Done
Run 1 (-j 16): 269.38 seconds / 13.36 kernels/hour [P:1389%, 168 maj. pagefaults]
Elapsed Time(E): 4:29.38 (269.38 seconds)
CPU usage (P): 1389%
Kernel time (S): 324.76 seconds
User time (U): 3419.04 seconds
Major page faults (F): 168
Minor page faults (R): 42275842
Context switches involuntarily (c): 995983
Context switches voluntarily (w): 84557
Run 2 (-j 16): 4/269.32 seconds / 13.37 kernels/hour [P:1389%, 99 maj. pagefaults]
Elapsed Time(E): 4:29.32 (269.32 seconds)
CPU usage (P): 1389%
Kernel time (S): 323.92 seconds
User time (U): 3417.69 seconds
Major page faults (F): 99
Minor page faults (R): 42283232
Context switches involuntarily (c): 984355
Context switches voluntarily (w): 82902
Run 3 (-j 16): 268.64 seconds / 13.40 kernels/hour [P:1393%, 166 maj. pagefaults]
Elapsed Time(E): 4:28.64 (268.64 seconds)
CPU usage (P): 1393%
Kernel time (S): 325.24 seconds
User time (U): 3417.64 seconds
Major page faults (F): 166
Minor page faults (R): 42298409
Context switches involuntarily (c): 969503
Context switches voluntarily (w): 83267Static linking [quqiu@quqiu-laptop dev]$ bash kcbench -d -j 16 -i 3
Processor: AMD Ryzen 7 5700U with Radeon Graphics [16 CPUs]
Cpufreq; Memory: powersave [amd-pstate-epp]; 15370 MiB
Linux running: 6.8.6-2-cachyos [x86_64]
Compiler: 阿菌•未霜 clang version 19.0.0git (https://github.com/llvm/llvm-project.git 0e44ffe817ae0f544199be70f468975fcc3ab5c5)
Linux compiled: 6.8.0 [/home/quqiu/.cache/kcbench/linux-6.8/]
Config; Environment: defconfig; CCACHE_DISABLE="1" LLVM="/home/quqiu/dev/amd64-kernel-arm_static/bin/"
Build command: make vmlinux
Filling caches: This might take a while... Done
Run 1 (-j 16): 214.00 seconds / 16.82 kernels/hour [P:1387%, 66 maj. pagefaults]
Elapsed Time(E): 3:34.00 (214.00 seconds)
CPU usage (P): 1387%
Kernel time (S): 269.55 seconds
User time (U): 2699.29 seconds
Major page faults (F): 66
Minor page faults (R): 35505008
Context switches involuntarily (c): 789551
Context switches voluntarily (w): 80684
Run 2 (-j 16): 213.77 seconds / 16.84 kernels/hour [P:1387%, 62 maj. pagefaults]
Elapsed Time(E): 3:33.77 (213.77 seconds)
CPU usage (P): 1387%
Kernel time (S): 267.14 seconds
User time (U): 2699.78 seconds
Major page faults (F): 62
Minor page faults (R): 35509845
Context switches involuntarily (c): 776327
Context switches voluntarily (w): 80221
Run 3 (-j 16): 213.78 seconds / 16.84 kernels/hour [P:1386%, 79 maj. pagefaults]
Elapsed Time(E): 3:33.78 (213.78 seconds)
CPU usage (P): 1386%
Kernel time (S): 268.23 seconds
User time (U): 2696.89 seconds
Major page faults (F): 79
Minor page faults (R): 35515431
Context switches involuntarily (c): 787074
Context switches voluntarily (w): 80407I use kcbench |
|
Not only is performance higher, when used with LLVM_TOOL_LLVM_DRIVER_BUILD, statically linked toolchain will even be smaller in size and faster to build than a dynamically linked toolchain, ultimately the entire host portion of the toolchain can be kept under 130MB. |
LLVM_TOOL_LLVM_DRIVER_BUILD |
I didn't observe any significant negative performance impact, and in my own use case the clang built this way was faster than neutron clang. (Also performed statically link+LTO+PGO+BOLT+mimalloc+2MiB alignment) |
Thank you for your answer This is my compilation parameter msg "Building LLVM..."
./build-llvm.py \
--projects clang lld polly \
--bolt \
--targets ARM AArch64 X86 \
--pgo llvm kernel-allmodconfig-slim \
--install-folder "installTmp" \
--vendor-string "Qiuqiu-$(date +%Y%m%d)" \
--defines LLVM_PARALLEL_COMPILE_JOBS=$(nproc --all) LLVM_PARALLEL_LINK_JOBS=$(nproc --all) ZLIB_LIBRARY=/usr/lib/libz.a LLVM_ENABLE_ZSTD=OFF CMAKE_EXE_LINKER_FLAGS="-static /usr/lib/libunwind.a /home/quqiu/dev/tc-build/clang/lib/clang/19/lib/x86_64-pc-linux-gnu/libclang_rt.builtins.a" LLVM_ENABLE_PIC=OFF CMAKE_BUILD_WITH_INSTALL_RPATH=1 LLVM_BUILD_STATIC=ON LIBCLANG_BUILD_STATIC=ON LLVM_LINK_LLVM_DYLIB=OFF LLVM_BUILD_LLVM_DYLIB=OFF CLANG_LINK_CLANG_DYLIB=OFF \
--show-build-commands \
--no-update |
|
My clang was built using other means, see This will only dynamically link to glibc and libstdc++ by default, if you build with fuchsia clang instead of system clang you can auto statically link to libc++, compiler-rt, and will only dynamically link to glibc, you can also add -DLLVM_BUILD_STATIC=ON to statically link to glibc, but I don't see any benefit to that, statically linking to LLVM itself and libc++ is already maximizing performance. For LLVM_TOOL_LLVM_DRIVER_BUILD, clang, lld, and llvm-tools are all symbolic links to "llvm", so in order to do representative PGO training, you need to make compiler and linker wrappers to ensure that only clang and lld can to output profraw. wrapper-clang: wrapper-ld: Then in the build script export LLVM_PROFILE_FILE=/dev/null, CC=wrapper-clang |
|
LTO(Full)+PGO+BOLT(Perf)+STATIC [quqiu@quqiu-laptop dev]$ bash kcbench -d -j 16 -i 3
Processor: AMD Ryzen 7 5700U with Radeon Graphics [16 CPUs]
Cpufreq; Memory: powersave [amd-pstate-epp]; 15370 MiB
Linux running: 6.8.6-2-cachyos [x86_64]
Compiler: 阿菌•未霜 clang version 19.0.0git (https://github.com/llvm/llvm-project.git deafb36f87a3541715854d4a620a4cfd6b1ac672)
Linux compiled: 6.8.0 [/home/quqiu/.cache/kcbench/linux-6.8/]
Config; Environment: defconfig; CCACHE_DISABLE="1" LLVM="/home/quqiu/dev/llvm-driver/bin"
Build command: make vmlinux
Filling caches: This might take a while... Done
Run 1 (-j 16): 213.56 seconds / 16.86 kernels/hour [P:1424%, 77 maj. pagefaults]
Elapsed Time(E): 3:33.56 (213.56 seconds)
CPU usage (P): 1424%
Kernel time (S): 246.05 seconds
User time (U): 2796.52 seconds
Major page faults (F): 77
Minor page faults (R): 36100892
Context switches involuntarily (c): 686395
Context switches voluntarily (w): 79233
Run 2 (-j 16): 214.22 seconds / 16.81 kernels/hour [P:1422%, 79 maj. pagefaults]
Elapsed Time(E): 3:34.22 (214.22 seconds)
CPU usage (P): 1422%
Kernel time (S): 246.64 seconds
User time (U): 2800.94 seconds
Major page faults (F): 79
Minor page faults (R): 36094329
Context switches involuntarily (c): 690408
Context switches voluntarily (w): 80015
Run 3 (-j 16): 214.21 seconds / 16.81 kernels/hour [P:1422%, 63 maj. pagefaults]
Elapsed Time(E): 3:34.21 (214.21 seconds)
CPU usage (P): 1422%
Kernel time (S): 246.98 seconds
User time (U): 2799.30 seconds
Major page faults (F): 63
Minor page faults (R): 36110063
Context switches involuntarily (c): 703722
Context switches voluntarily (w): 79644 |
As part of putting a build of clang up on kernel.org, statically linking all dependencies would simplify distribution for the various linux distros (I suspect). I don't know how to do this today in LLVM's cmake; maybe we need to add some things to upstream LLVM to do so.
The text was updated successfully, but these errors were encountered: