[server] Use the multiprocess library on every platform
#4095
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vodorok
added
CLI 💻
whisperity
requested changes
Nov 22, 2023
whisperity
requested changes
Nov 22, 2023
|
Did some tests on windows, and I could use the multiprocess lib without issues. I will remove the branching completely. |
vodorok
force-pushed
the
osx_multiproc_store
branch
3 times, most recently
from
8d12a44
to
12888c1
Compare
bruntib
approved these changes
Nov 27, 2023
@vodorok Let's just make sure these claims hold true for sufficiently old Python versions such as 3.9. Other than that, I think this is good to go. |
whisperity
reviewed
Dec 5, 2023
whisperity
reviewed
Dec 6, 2023
whisperity
changed the title
Even though we swapped a higher level interface to a lower level (ProcessPoolExecutor -> Multiprocess.Pool), it is worth it because on all 3 systems the multiprocess handling is the same. Multiprocessing is broken on osx, (probably because of the fork -> spawn change.) This patch intends to correct it by using the newly added multiprocess library. On Windows only single threaded execution is supported. With the introduction of the multiprocess lib multiprocessing is now available.
vodorok
force-pushed
the
osx_multiproc_store
branch
from
12888c1
to
f8e6f56
Compare
whisperity
approved these changes
Dec 7, 2023
|
Do not merge yet |
if sys.platform in ["darwin", "win32"]:
from multiprocessing import Pool as MultiProcessPool
elif sys.platform == ":
from concurrent.futures import ProcessPoolExecutor as MultiProcessPool
from codechecker_common.multiprocessing import MultiProcessPool
# ...
with MultiProcessPool(...) as pool:
...(..., pool.map) |
|
Made a different implementation where the linux platform still uses the ProcessPoolExecutor class. |
vodorok
changed the title
vodorok
changed the title
vodorok
changed the title
whisperity
changed the title
multiprocess library on every platform
whisperity
added
dependencies 📦
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Multiprocessing is broken on osx, (probably because of the fork -> spawn change). This patch intends to correct it by using the newly added multiprocess library.