-
Notifications
You must be signed in to change notification settings - Fork 35.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kernel: Streamline util library #29015
base: master
Are you sure you want to change the base?
Conversation
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code CoverageFor detailed information about the code coverage, see the test coverage report. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
Is the idea here that |
Updated 0e64745 -> 778c021 ( re: #29015 (comment)
I do think that's generally a good thing to do, but it's not really the goal of this PR. The main thing this PR is trying to do is provide a better kernel API and reduce the size of the kernel library. Because libbitcoin_kernel depends on libbitcoin_util, any external application using the kernel will also depend on util. So we should try to avoid including functions and types in util that are not likely to be useful to external applications, or are unstable and meant for internal use. The common library is usually a better place for these things. |
Does that mean that things should only go in util if they're going to be used by the kernel library then? That would presumably mean util/bitdeque.h and util/sock.cpp should also eventually go elsewhere, for example? That seems like a bad approach to me (in that it means code has to move into and out of util based on whether it's relevant to the kernel, rather than due to any changes to the code itself, and users then potentially have to deal with it moving into/outof the util:: namespace). |
No, I don't think so, and I don't think much else is going to move after this PR. I think only code that shouldn't be used by the kernel or kernel applications should be moved out of util, not just code that isn't used by the kernel. I agree just moving any code that isn't used by the kernel would be a bad approach, and wrote basically the same comment as you recently in #27989 (comment), so I think we are in agreement. |
Updated 778c021 -> 8b21f41 ( |
8748d63
to
b234094
Compare
Does it make sense to improve https://github.com/bitcoin/bitcoin/blob/master/doc/design/libraries.md by expressing the idea from the quotes above more clearly? As for now, it reads:
|
This seems like the right place to talk about the relationship between the util and kernel library. My work on the kernel library has been focusing on reducing its essence to validation code, pruning external dependencies, and stripping out as much system related functionality as possible. I think if the goal is to have the kernel be capable of running on as many platforms as possible, potentially beyond the number of platforms that Bitcoin Core is capable of today and extending into embedded environments, this last point becomes crucial. Looking at the content of In my opinion, making the The asterisk here is that some of the utilities that are not currently used by the kernel do not introduce new dependencies, since they entirely rely on code that is already used by the kernel. I also generally think that some utilities could remain in the util library even though they are not currently used by validation code. The criteria I propose for evaluating if modules should remain in util after this PR are:
Looking at @hebasto's comment here, Between this PR and #28690 I think it would be nice if common criteria for |
Would it make sense to just merge common into util; but have a "stripped" version of util available for the kernel, that excludes stuff that doesn't match the 5 points above? That way the difference is just at the build system level, it doesn't involve moving files around or moving code between namespaces... Then the advice would be: (a) just put things in util, (b) but only add things to the "util-core" section of the build file when they're needed by the kernel; (c) keep things in the util:: namespace; (d) split stuff into different .h/.cpp files pretty aggressively? That would imply a different approach for the util: move fees.h and error.h to common/messages.h commit here. |
re: #29015 (comment)
Idea to build util and util-core (or util-lite?) libraries is interesting, because then it would be easy to add/remove things from util library depending on whether the kernel needs them, and it would never require renaming anything. I could see this being useful in the future if util library started growing a lot and accumulating more features of over time. But for now, IMO, just sticking with the current util/common separation and using the cleanup in this PR would organize things better and be less confusing than having two util versions, and it would require moving fewer things. Even though the libraries.md document is currently ambiguous, I think it should be pretty easy to remove the ambiguity here. re: #29015 (comment)
I think these are good rules if you replace "basic data structure utilities" with "basic utilities that to do not require outside dependencies", and also drop the 3rd rule. It seems like with the 3rd rule you are trying to make a distinction between "data structure utilities" and "system-related utilities", and I don't think it helps to draw a dividing line there. |
Added 1 commits b234094 -> 6d8cb4d ( re: #29015 (comment)
Just to address these points directly, I think the goals of making the kernel portable and working on any platform, and making the kernel minimal working and in constrained environments are great goals. I just don't you need to |
One of the guiding principles for the kernel project so far has been to prohibit future re-couplings of decoupled modules. The tracking issue says "Any further coupling of our consensus engine with non-consensus modules will result in linker errors, preventing this project from becoming a Sisyphean task of battling coupling regressions.". How could this fit in with leaving unused content in the library? |
I reread the issue you linked to, but I don't understand what it implies here. I do think you can rely on linker errors to ensure kernel code isn't calling wallet code or GUI code or P2P code or storing state in global variables. But what else do you want beyond that? And how does removing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, I think I'm on roughly the same page:
- keeping Split in util make sense
- moving our string stuff into the util namespace seems fine * moving the spanparsing stuff to script mostly makes sense -- though I feel a bit like mixing consensus critical script manipulation and miniscript/descriptor parsing in the same namespace is a bit clunky?
- moving fees.h/error.h to "common" makes sense; though if we can make "common" just be a broader part of "util" that seems better long term.
- moving TransactionError into node rather than common doesn't make sense to me though
doc/design/libraries.md
Outdated
@@ -87,14 +88,15 @@ class bitcoin-qt,bitcoind,bitcoin-cli,bitcoin-wallet bold | |||
|
|||
- *libbitcoin_consensus* should be a standalone dependency that any library can depend on, and it should not depend on any other libraries itself. | |||
|
|||
- *libbitcoin_util* should also be a standalone dependency that any library can depend on, and it should not depend on other internal libraries. | |||
- *libbitcoin_crypto* should be a standalone dependency that any library can depend on, and it should not depend on any other libraries itself. (It is not shown in diagram above to save space, but has the exact same connections as *libbitcoin_consensus*, which is shown.) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't libconsensus necessarily depend on libcrypto (for sha256 at least)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re: #29015 (comment)
Doesn't libconsensus necessarily depend on libcrypto (for sha256 at least)?
Yes that should be right. I think I just deluded myself because I didn't want to update the mermaid diagram. Updated now though.
Maybe removing |
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the Possibly this is due to a silent merge conflict (the changes in this pull request being Leave a comment here, if you need help tracking down a confusing failure. |
There seems to be a conflict with #30098. Maybe rebase? |
Thanks! Rebased now.
Yes this could be a good idea. The script started out just being a few lines of bash listing symbols exported by one library and imported by another one. But then it grew as it was extended to support multiple libraries and suppress known errors. So now it's no longer a small script, though it is pretty clean and well commented. Other possible followups besides translating the script could be:
Rebased 7689dfd -> 1c26d10 ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re-ACK 1c26d10
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK 1c26d10.
My Guix build:
|
🚧 At least one of the CI tasks failed. Make sure to run all tests locally, according to the Possibly this is due to a silent merge conflict (the changes in this pull request being Leave a comment here, if you need help tracking down a confusing failure. |
CI failure suggests there's a silent merge conflict. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re-ACK c7376ba
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
re-ACK c7376ba.
My Guix build:
x86_64
f33ae90e6d190245d34712f544cd458f282a2c90890029dfbd88d74e8823fb50 guix-build-c7376babd19d/output/aarch64-linux-gnu/SHA256SUMS.part
ee444d54e0b9cebfa6ab0c91fc8b543d414a98338d46d299f3b302273139c81f guix-build-c7376babd19d/output/aarch64-linux-gnu/bitcoin-c7376babd19d-aarch64-linux-gnu-debug.tar.gz
fdc7400c04fd6bfec5d0ab3148b0f4da110974ecda0859184adad29f38422f6f guix-build-c7376babd19d/output/aarch64-linux-gnu/bitcoin-c7376babd19d-aarch64-linux-gnu.tar.gz
9ef394d81127cf5ee2eb64508feb0a96f72763073fe0ed9a0901d6008e51b2b8 guix-build-c7376babd19d/output/arm-linux-gnueabihf/SHA256SUMS.part
40cec39c64338a03e7553bb9734d48061d5d1669ab6a59d6250d0dcd5821fde0 guix-build-c7376babd19d/output/arm-linux-gnueabihf/bitcoin-c7376babd19d-arm-linux-gnueabihf-debug.tar.gz
acc8be16daaecab8fdf96763973b8fe7e3e5e9e8014e1e04cf9133cba9f55375 guix-build-c7376babd19d/output/arm-linux-gnueabihf/bitcoin-c7376babd19d-arm-linux-gnueabihf.tar.gz
791971a7afaef6827d3a2b4da5f92a10aa28ff906efd372ef524b900c6815ba7 guix-build-c7376babd19d/output/arm64-apple-darwin/SHA256SUMS.part
bad59a9c07844fa7b788a8b17063c950fa6e124e046d1bee02959344438350dd guix-build-c7376babd19d/output/arm64-apple-darwin/bitcoin-c7376babd19d-arm64-apple-darwin-unsigned.tar.gz
41de8c1590fc2479ac52946f61affcc592fa7c187abcef02972933e91487ff73 guix-build-c7376babd19d/output/arm64-apple-darwin/bitcoin-c7376babd19d-arm64-apple-darwin-unsigned.zip
b4e834ae5e563d3cf975a0bc5cae1ba9734b19df387eddc45677f910b8bb5043 guix-build-c7376babd19d/output/arm64-apple-darwin/bitcoin-c7376babd19d-arm64-apple-darwin.tar.gz
5d2928dc2f3b6c5390a2020221f1f069bb4c0b0acff09c0d2c79b35ef486b5c9 guix-build-c7376babd19d/output/dist-archive/bitcoin-c7376babd19d.tar.gz
4b8e4f59977136e23ae0dc20816a671257cee4156d80551cbe1c27024857e8f2 guix-build-c7376babd19d/output/powerpc64-linux-gnu/SHA256SUMS.part
d5ce1a3593408c9eb7d1f2fc3d61e7d400381335faeca5cc6be11999c798e4c6 guix-build-c7376babd19d/output/powerpc64-linux-gnu/bitcoin-c7376babd19d-powerpc64-linux-gnu-debug.tar.gz
e11a741a82205faa1c271aa62f5a20329cf46c30db46bda37ed25dc83cbca662 guix-build-c7376babd19d/output/powerpc64-linux-gnu/bitcoin-c7376babd19d-powerpc64-linux-gnu.tar.gz
6ddd58c3fe5e4c600a65b5b9e4bb7e0b99bd3cd5f8655366bb0e29e498e480fe guix-build-c7376babd19d/output/riscv64-linux-gnu/SHA256SUMS.part
c35ab4f03e23c3889fac7c1a3a64d979af00cb762f6425599149249e8df7e4fd guix-build-c7376babd19d/output/riscv64-linux-gnu/bitcoin-c7376babd19d-riscv64-linux-gnu-debug.tar.gz
f46bee10789cb0d2f1df0a386b93ebf92ecbd9faaf3446b2df8dc1c5fa77e46c guix-build-c7376babd19d/output/riscv64-linux-gnu/bitcoin-c7376babd19d-riscv64-linux-gnu.tar.gz
5ba9da03a94da08d34d20edff2d9dd1de729f4087897d3573d5a17fbb053f0d7 guix-build-c7376babd19d/output/x86_64-apple-darwin/SHA256SUMS.part
4b0a6d20c60ff4bfb85bb143b13b0ad28f9c65d536cdbda96b695abf03a0d268 guix-build-c7376babd19d/output/x86_64-apple-darwin/bitcoin-c7376babd19d-x86_64-apple-darwin-unsigned.tar.gz
9525e7082ae7c691381af6abff1a1733b7a2efd772ffbe9bbd98cb522cd8609d guix-build-c7376babd19d/output/x86_64-apple-darwin/bitcoin-c7376babd19d-x86_64-apple-darwin-unsigned.zip
a62eb1faf7a413267aab397e0c687016a2055f225f5a07d2780eaa7678e0260e guix-build-c7376babd19d/output/x86_64-apple-darwin/bitcoin-c7376babd19d-x86_64-apple-darwin.tar.gz
70ec027a5d59eefeedf1943c96504d39739753f367442211d0353512b1ac798d guix-build-c7376babd19d/output/x86_64-linux-gnu/SHA256SUMS.part
793c670bbd8724271c4966cbcffae13a9c876821aeee9ede7449e705acd463e7 guix-build-c7376babd19d/output/x86_64-linux-gnu/bitcoin-c7376babd19d-x86_64-linux-gnu-debug.tar.gz
4c711b6709ee78686f83296bb1e2596772d103cf4fc9959b50085a7d62652921 guix-build-c7376babd19d/output/x86_64-linux-gnu/bitcoin-c7376babd19d-x86_64-linux-gnu.tar.gz
3562a253c9bc448238aad4bb612c73f4fef6899ef297967b4a77b46fab40e73d guix-build-c7376babd19d/output/x86_64-w64-mingw32/SHA256SUMS.part
3ae8eb90a5dd6fb50caf0f3df16bc0cb60646d1a126a45b89a70bc5d12d00ec4 guix-build-c7376babd19d/output/x86_64-w64-mingw32/bitcoin-c7376babd19d-win64-debug.zip
23f8f28cc9bb4c63d4c4420d8ccf5c0b2547c34b0ebf4b73b10d3553e7fc3681 guix-build-c7376babd19d/output/x86_64-w64-mingw32/bitcoin-c7376babd19d-win64-setup-unsigned.exe
c709e3dbcf3267a6f18ca984b8308bf7f05d58ddbbae83ae9e024db84693a861 guix-build-c7376babd19d/output/x86_64-w64-mingw32/bitcoin-c7376babd19d-win64-unsigned.tar.gz
2cf7011773d95db8398fdb538a5650eb3abd443c2fc18ea091ab07059c778eaf guix-build-c7376babd19d/output/x86_64-w64-mingw32/bitcoin-c7376babd19d-win64.zip
Remove
fees.h
,errors.h
, andspanparsing.h
from the util library. Specifically:Split
functions fromutil/spanparsing.h
toutil/string.h
, usingutil
namespace for clarity.script/parsing.h
since they are used for descriptor and miniscript parsing.util/fees.h
andutil/errors.h
intocommon/messages.h
so there is a place for simple functions that generate user messages to live, and these functions are not part of the util library.Motivation for this change is that the util library is a dependency of the kernel, and we should remove functionality from util that shouldn't be called by kernel code or kernel applications. These changes should also improve code organization and make functions easier to discover. Some of these same moves are (or were) part of #28690, but did not help with code organization, or made it worse, so it is better to move them and clean them up in the same PR so code only has to change one time.