build: LLD based macOS toolchain #21778
Open
+66
−162
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
|
EDIT: Just a misunderstanding |
I might be misunderstanding your comment, but the opposite is happening here. We're migrating away from using Apples, irregularly open sourced tools (cctools, ld64), to just vanilla LLVM (lld), which is entirely open source. |
My bad! I thought this was switching from an unsupported apple open-source toolchain to an supported apple closed source toolchain, not to a vanilla toolchain. My blinding hate of apple has kept me from seeing reality! Thanks for the clarification. This all seems very good. |
|
Very nice! Strong concept ACK, but obviously we can't use this until all features we need are supported. But it's great to see that linking finally works! For a little extra context beyond what @fanquake said above, the macOS clang+cctools build has always been a wonky outlier because we have to build our own (very unsupported) toolchain first. LD64 is apple's libre/open-source linker, and there is a community project that maintains a build environment for cross builds. For a few years, there has been slow development on a darwin functionality for llvm's lld, but it's only really picked up speed and become usable in the last year or two. This PR would bring our Apple builds much more inline with others in that it would finally be using a supported libre/open-source compiler/linker combo. @fanquake Is cctools still needed at all? LLVM ships bitcode equivalents of most binutils tools (ar, nm, ranlib, etc), I wonder if we can use those instead. Maybe they require lto? |
@theuni no cctools is no longer required. I drop the native_cctools as part of this PR and we just use the LLVM tools. |
Apologies for reading too quickly, I see now that I only looked at the first commit. Turns out the 3rd one is the one I expected :) |
|
Rebased for #21793, and rebased on top of #19817. Made some additional changes, such as renaming the Note: I'm pretty sure the use of |
fanquake
force-pushed
the
llvm_13_ld64_lld
branch
from
b79ed8c
to
f068682
Compare
This was referenced Jul 13, 2021
fanquake
force-pushed
the
llvm_13_ld64_lld
branch
from
f068682
to
e6b20d5
Compare
fanquake
force-pushed
the
llvm_13_ld64_lld
branch
7 times, most recently
from
e1b2e6e
to
1179221
Compare
|
Guix build (aarch64): Looks like there is a mismatch on the arm64 build. |
fanquake
force-pushed
the
llvm_13_ld64_lld
branch
from
f58a867
to
e208d5d
Compare
Disabled adhoc-codesigning for now, as the non-determinism is coming from the identifier field (also why it only happens for arm64). also cc @theuni |
|
Guix Build (aarch64, x86_64): a0f00b836d0843e3e9be0cef471f455d113de49da467424718fafeb981490be5 guix-build-e208d5df7937/output/aarch64-linux-gnu/SHA256SUMS.part
a1ccae9ef1a5f4f1641c864c2aa173c7fde2f63de5519c8a13bf2e13b1a16b94 guix-build-e208d5df7937/output/aarch64-linux-gnu/bitcoin-e208d5df7937-aarch64-linux-gnu-debug.tar.gz
79d2affa00eda5657e322b864fe31c49cde9971c5a1ef303fa781d4740748979 guix-build-e208d5df7937/output/aarch64-linux-gnu/bitcoin-e208d5df7937-aarch64-linux-gnu.tar.gz
f8ca89361fcb1b0958756c153ac434b7b7f93b1a95136746367d648e33a017ea guix-build-e208d5df7937/output/arm-linux-gnueabihf/SHA256SUMS.part
2ae9e393b8d363de64226b765bf620851bc58e1dc0409faf3aa9342f1c91d7e4 guix-build-e208d5df7937/output/arm-linux-gnueabihf/bitcoin-e208d5df7937-arm-linux-gnueabihf-debug.tar.gz
63955c4254a76f554dcb2574eb5d0eea7827360199ba8994544a7f1e73f2d7b8 guix-build-e208d5df7937/output/arm-linux-gnueabihf/bitcoin-e208d5df7937-arm-linux-gnueabihf.tar.gz
5bcde35563e11ae191ec424bb43227f76c91d5819fdf336cb679763fb4f37b08 guix-build-e208d5df7937/output/arm64-apple-darwin/SHA256SUMS.part
9b3f1fb66cf2956d7dd65e9beaa6221d37d8070b08a0f839b80d48c183901c96 guix-build-e208d5df7937/output/arm64-apple-darwin/bitcoin-e208d5df7937-arm64-apple-darwin-unsigned.tar.gz
b51bbc77fdaf6db5d460450b8f93a9b5f9d225e915faaa055372778fe5b285ac guix-build-e208d5df7937/output/arm64-apple-darwin/bitcoin-e208d5df7937-arm64-apple-darwin-unsigned.zip
133ebf8255bc48ab79d79ee03c3cf797f85d61867d5f38c3fce5a11d284ca696 guix-build-e208d5df7937/output/arm64-apple-darwin/bitcoin-e208d5df7937-arm64-apple-darwin.tar.gz
5d2ad53e635b3ec0917a74627a8bacfd3efccd3dc2b05268d012a506c56f5ca6 guix-build-e208d5df7937/output/dist-archive/bitcoin-e208d5df7937.tar.gz
df2f04b6139c25d7af8e6d6a1d126a35aeab599b2ae948c4084d7a1e9abe5378 guix-build-e208d5df7937/output/powerpc64-linux-gnu/SHA256SUMS.part
438bd47ffadea4967f9a4542898d8ff931d0ae27bb4f535631362224409ba2cd guix-build-e208d5df7937/output/powerpc64-linux-gnu/bitcoin-e208d5df7937-powerpc64-linux-gnu-debug.tar.gz
4052325b00ede1c5a1b5b5896d55e89abc72216ede2e25f4cf6da62873a0f30a guix-build-e208d5df7937/output/powerpc64-linux-gnu/bitcoin-e208d5df7937-powerpc64-linux-gnu.tar.gz
5bbbf8009e3e1f4ed7f058dc562b3e3c64613e45c8ddf92049e205978cb6e540 guix-build-e208d5df7937/output/riscv64-linux-gnu/SHA256SUMS.part
9cb924b9826be7738b604e86150b667e4a52d0771e87e59337ba9b1a1fdef18e guix-build-e208d5df7937/output/riscv64-linux-gnu/bitcoin-e208d5df7937-riscv64-linux-gnu-debug.tar.gz
19c50189af630703c45a42bf567cea7334c142315d4769de6e613b97a9079942 guix-build-e208d5df7937/output/riscv64-linux-gnu/bitcoin-e208d5df7937-riscv64-linux-gnu.tar.gz
8e17100eed1dd9f37667ea91163006ad93d5abfb718d423e90a93810465d19a0 guix-build-e208d5df7937/output/x86_64-apple-darwin/SHA256SUMS.part
1c4c1e764d199b404d29923724339e518973b9861f1bf2ee3c58fe67ef4f91f1 guix-build-e208d5df7937/output/x86_64-apple-darwin/bitcoin-e208d5df7937-x86_64-apple-darwin-unsigned.tar.gz
916ed3b18162215be8f331e521170fca10afeb03d0348aad00731b419351ae11 guix-build-e208d5df7937/output/x86_64-apple-darwin/bitcoin-e208d5df7937-x86_64-apple-darwin-unsigned.zip
6caf13d5bc229b5017bc0292586e79d751917410c7d39889db39b97ac70b6597 guix-build-e208d5df7937/output/x86_64-apple-darwin/bitcoin-e208d5df7937-x86_64-apple-darwin.tar.gz
45c74fc8b64f808d27836c8b12fe433447606fbc8e06f42825003fe0ab9e963b guix-build-e208d5df7937/output/x86_64-linux-gnu/SHA256SUMS.part
60f9182ff31c5ac33f4b22c8150146185c66368ca0524809879bfcf7faca9b4c guix-build-e208d5df7937/output/x86_64-linux-gnu/bitcoin-e208d5df7937-x86_64-linux-gnu-debug.tar.gz
e7e4b0a37766a891d2c63a9fca5e9f3694e9af3d67a08726cb036ddf1ab21bc2 guix-build-e208d5df7937/output/x86_64-linux-gnu/bitcoin-e208d5df7937-x86_64-linux-gnu.tar.gz
be58ad405308a41a75c8d4901737ec8875de009e7d3b29b40913d3c7452cc580 guix-build-e208d5df7937/output/x86_64-w64-mingw32/SHA256SUMS.part
78974f958c83411cab5404703ec47b83b97ba636af3761293970960cfe08064d guix-build-e208d5df7937/output/x86_64-w64-mingw32/bitcoin-e208d5df7937-win64-debug.zip
fb10861ab7f7c9b13255660dde2f7b8ad8fab9cebe4e1324aaea052364fe8a7d guix-build-e208d5df7937/output/x86_64-w64-mingw32/bitcoin-e208d5df7937-win64-setup-unsigned.exe
0b4cc5f9702c2f3f33852097c83fa81c26589aa7cd24bc3257527218fbd86943 guix-build-e208d5df7937/output/x86_64-w64-mingw32/bitcoin-e208d5df7937-win64-unsigned.tar.gz
d158efd3734dc549be953ff5ca662eff479e8b2d7f7eeef08eb13f4b442ce82d guix-build-e208d5df7937/output/x86_64-w64-mingw32/bitcoin-e208d5df7937-win64.zip |
theuni
reviewed
May 21, 2024
|
It's not clear to me if dsymutil ends up being used as part of our build at all. If so we should fix depends, if not we should remove it. |
fanquake
force-pushed
the
llvm_13_ld64_lld
branch
from
e208d5d
to
b32b3a1
Compare
I think it might actually be unused at this point, but also have a vauge memory of a Qt related failure, if it's missing.. Have pushed up a change to have it set properly for now. |
theuni
reviewed
May 21, 2024
Not supported by llvm-ar / llvm-ranlib.
Only build ld64 from cctools. Disable adhoc codesigning, to avoid non-determinism.
Adjust the security check for: ld64.lld: warning: Option `-allow_stack_execute' is not yet implemented. ld64.lld: error: -fixup_chains is incompatible with -no_pie and to account for the embedding of LLVMs version number.
LLD enables ZERO_AR_DATE by default, setting it to zero would enable non-determinism, setting it to any other value is ignored. See: https://github.com/llvm/llvm-project/blob/main/lld/docs/MachO/ld64-vs-lld.rst.
fanquake
force-pushed
the
llvm_13_ld64_lld
branch
from
b32b3a1
to
e8c25e8
Compare
|
Guix build (aarch64): Another mismatch, this time on riscv64 -_- |
I've just rebuilt on 2 different machines (one aarch64 one x86_64), and they match your output here (removing my comment above): 00becde2dd12878e3b9f50f27899a6a8b752343dade7c71781632715c3001473 guix-build-e8c25e8a35e3/output/aarch64-linux-gnu/SHA256SUMS.part
a685b9cee54014e74639be1e8db2d55b7c008fdb3b31c1c708c364a49b56759a guix-build-e8c25e8a35e3/output/aarch64-linux-gnu/bitcoin-e8c25e8a35e3-aarch64-linux-gnu-debug.tar.gz
d61228158409802e5aef11c39a0da5653a6c7e870d5f500483c32c75f319e8b6 guix-build-e8c25e8a35e3/output/aarch64-linux-gnu/bitcoin-e8c25e8a35e3-aarch64-linux-gnu.tar.gz
49447a196ead1f153cbebab2e22d229cb48f2736eec5c912c2ec5cebf8a58cad guix-build-e8c25e8a35e3/output/arm-linux-gnueabihf/SHA256SUMS.part
1e4510f6a5145810577b8ebc2eb5f38b3d2c28845e9d84ffe000d8cbb82429fa guix-build-e8c25e8a35e3/output/arm-linux-gnueabihf/bitcoin-e8c25e8a35e3-arm-linux-gnueabihf-debug.tar.gz
a3c014032ac22c2fc0034a5708b157884395f37f3d20dfa6070cfb2a6aeebdf5 guix-build-e8c25e8a35e3/output/arm-linux-gnueabihf/bitcoin-e8c25e8a35e3-arm-linux-gnueabihf.tar.gz
6377bf759602fb5dc00d7992b126ddc75cf03d71ce259c76c509299dac51be52 guix-build-e8c25e8a35e3/output/arm64-apple-darwin/SHA256SUMS.part
7191d5a512ac9b7e934e771985eb2743180a00285b3e5275de1ffdc1e504e68e guix-build-e8c25e8a35e3/output/arm64-apple-darwin/bitcoin-e8c25e8a35e3-arm64-apple-darwin-unsigned.tar.gz
73516bfd8acb17f865b53e67e421dccfd0644c6582f462d8ebe710291c2b5367 guix-build-e8c25e8a35e3/output/arm64-apple-darwin/bitcoin-e8c25e8a35e3-arm64-apple-darwin-unsigned.zip
07cc918282582193d8745cc7dcd4a4a4474e38d079b0d2831efdef59cb322449 guix-build-e8c25e8a35e3/output/arm64-apple-darwin/bitcoin-e8c25e8a35e3-arm64-apple-darwin.tar.gz
edcc1c61aa12b78c7a8e87d141832891e05f48b4a687ed7e3fae668236f339ff guix-build-e8c25e8a35e3/output/dist-archive/bitcoin-e8c25e8a35e3.tar.gz
29601464afa40104ad8c6b4b10a5fde0649aba6f4bfea1b93258b7bfcaab7a5a guix-build-e8c25e8a35e3/output/powerpc64-linux-gnu/SHA256SUMS.part
65c308cddf5cbac4132104a5b4d6cf7ec85e88b4f967c66cd942742996def56a guix-build-e8c25e8a35e3/output/powerpc64-linux-gnu/bitcoin-e8c25e8a35e3-powerpc64-linux-gnu-debug.tar.gz
513c91b558e52ac27e6e33d916bfc378a6be6b2b218fb37712f05a44746fb68d guix-build-e8c25e8a35e3/output/powerpc64-linux-gnu/bitcoin-e8c25e8a35e3-powerpc64-linux-gnu.tar.gz
72c790770e49bacd4e66eb2e93f6a3a2dc596c36254bf1ec00775bbd60121c31 guix-build-e8c25e8a35e3/output/riscv64-linux-gnu/SHA256SUMS.part
c7921f8f5bfe5c463e266b24399e600c2b67552a50dbf1414cacb5805271c561 guix-build-e8c25e8a35e3/output/riscv64-linux-gnu/bitcoin-e8c25e8a35e3-riscv64-linux-gnu-debug.tar.gz
4f106246e1184a32289d2ba5251fc00af852e38351e45b3bcd974d1b099e515b guix-build-e8c25e8a35e3/output/riscv64-linux-gnu/bitcoin-e8c25e8a35e3-riscv64-linux-gnu.tar.gz
4e26090aa5ffb709088cb4e4e6aba32c0c2017a1fa776f5ec4d7be36e3ccb8aa guix-build-e8c25e8a35e3/output/x86_64-apple-darwin/SHA256SUMS.part
87282cc09b9c57416c8f92ff5ae2614c9a9d26fdcfe3c16931df0fdedae9e1b4 guix-build-e8c25e8a35e3/output/x86_64-apple-darwin/bitcoin-e8c25e8a35e3-x86_64-apple-darwin-unsigned.tar.gz
25fd762a3a7b0d81f738c3b63272188094d42139b39c6f8dd92c30c54ccc86be guix-build-e8c25e8a35e3/output/x86_64-apple-darwin/bitcoin-e8c25e8a35e3-x86_64-apple-darwin-unsigned.zip
d8c01baf156546546620d0ab17933478772527be4701b7479881a5547c527e2a guix-build-e8c25e8a35e3/output/x86_64-apple-darwin/bitcoin-e8c25e8a35e3-x86_64-apple-darwin.tar.gz
b4eafb5e1c6730105e47cd7cd6ede361a4487250c903661c0ccebce8eeda5898 guix-build-e8c25e8a35e3/output/x86_64-linux-gnu/SHA256SUMS.part
819e54887b0eb2d659dccfbfcec9a1d49e47914ce46baae3b52b2821a844098a guix-build-e8c25e8a35e3/output/x86_64-linux-gnu/bitcoin-e8c25e8a35e3-x86_64-linux-gnu-debug.tar.gz
25e48ea3a8f2d86ebea8d8ea96a902f3494197e93a65d4ff986d37506771a43a guix-build-e8c25e8a35e3/output/x86_64-linux-gnu/bitcoin-e8c25e8a35e3-x86_64-linux-gnu.tar.gz
d73e31db2488cf422704e3abf432bb430f9155b8aafd4a9a501f826785cbe1d9 guix-build-e8c25e8a35e3/output/x86_64-w64-mingw32/SHA256SUMS.part
23818afba6bbd8de1c2c2d6d6b64c05d87400d424f4c9db516b854c75b29f50a guix-build-e8c25e8a35e3/output/x86_64-w64-mingw32/bitcoin-e8c25e8a35e3-win64-debug.zip
facdda5d82b05091bb98419c7b6913e4d0019c58f4d67f95d750fc632e32c611 guix-build-e8c25e8a35e3/output/x86_64-w64-mingw32/bitcoin-e8c25e8a35e3-win64-setup-unsigned.exe
45a1252ce93fb0d8e1ebde66c16542c1bbc8b6efd6280ec7f1553415f80461fd guix-build-e8c25e8a35e3/output/x86_64-w64-mingw32/bitcoin-e8c25e8a35e3-win64-unsigned.tar.gz
8b4bc3b6cd78713e045db74eed4b3fbd711dd7e09980da386100e72dce33097e guix-build-e8c25e8a35e3/output/x86_64-w64-mingw32/bitcoin-e8c25e8a35e3-win64.zip |
Thanks, can confirm the correct one is picked up by our configure now. |
|
Tested e8c25e8: the guix build, a local depends build and a local normal build on Intel macOS 14.5. Tested a normal build on macOS 13.6.7. Guix hashes (Ubuntu, AMD), matches what @TheCharlatan and @fanquake found above. |
|
I presume macos-cross compilation on risv64 metal still fails? If it is expected to pass now, I can re-try this. |
Quite possible it'll pass now. Given the previous failure was in compiling cctools. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This switches us to using a LLD based toolchain for macOS builds.
Benefits