Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Windows build doesn't set parallel build thread count properly #2956

Open
NeedsMoar opened this issue May 1, 2024 · 2 comments
Open

Comments

@NeedsMoar
Copy link

What Operating System(s) are you seeing this problem on?

Windows

dlib version

19.24.2

Python version

3.11.7

Compiler

MSVC 19.39.33523

Expected Behavior

Build should set appropriate parallel thread count

Current Behavior

Build only uses 2 threads on 32/64 core machine with 512GB of ram.

Steps to Reproduce

pip install dlib
observe very long build time

Anything else?

A fix for this should be a quick conditional in the num_available_cpu_cores(ram_per_build_process_in_gb) function in setup.py if the os is Windows, something like:

try: 
   if platform.system() == "Windows":
      import win32.win32api as w32api
      memstatus = w32api.GlobalMemoryStatusEx()
      mem_bytes = memstatus['AvailPhys']
      mem_gib = mem_bytes/(1024.**3)
    else:
      #snip current sysconf stuff
   num_cores = multiprocessing.cpu_count() 
   #snip calculations

The windows functions should be standard with win python installs but I didn't submit a pull request since I haven't tested this inside that file, I just know the given functions to determine memory work.

The AvailPhys field returned doesn't include pagefile or virtual memory so there shouldn't be any chance of hitting either of those.
Right now it fails and returns 2 because os.sysconf isn't defined on Windows so it throws the exception and uses the default value.

@davisking
Copy link
Owner

Yeah, that's a good idea. Send me a PR that works for you on your windows machine :)

@NeedsMoar
Copy link
Author

Sorry, missed this. I'll whip one up for you soonish, shouldn't take long since that's pretty much all of it up there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants