Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify default n_workers for postprocessing in documentation #225

Closed
mleachNREL opened this issue Apr 26, 2021 · 5 comments · Fixed by #229
Closed

Specify default n_workers for postprocessing in documentation #225

mleachNREL opened this issue Apr 26, 2021 · 5 comments · Fixed by #229

Comments

@mleachNREL
Copy link

Default value for n_workers is 2 (despite the fact that it can be set to 32 for Eagle runs). This default value should be reported in the documentation. Thanks!

@bonnema
Copy link
Contributor

bonnema commented Apr 26, 2021

36 I think...

@nmerket
Copy link
Member

nmerket commented May 3, 2021

For clarification, that's for how many 36 core nodes we add as workers to the dask cluster. So, for example, n_workers = 2 gives you 72 cores worth of compute. It's a little more complicated than that because we intentionally don't utilize all the cores on those machines because we're more memory constrained than compute constrained.

But, yes, that default should be in the docs.

@bonnema
Copy link
Contributor

bonnema commented May 3, 2021

For clarification, that's for how many 36 core nodes we add as workers to the dask cluster. So, for example, n_workers = 2 gives you 72 cores worth of compute. It's a little more complicated than that because we intentionally don't utilize all the cores on those machines because we're more memory constrained than compute constrained.

But, yes, that default should be in the docs.

What about trying to use the big mem nodes?

@nmerket
Copy link
Member

nmerket commented May 3, 2021

@bonnema Mostly because the demand for time on those nodes is crazy. It often takes longer to wait for a bigmem node to become available than for the entire rest of the batch to run. We do have options to through the eagle.postprocessing.node_membory_mb field to specify a bigmem node if that's what you want, but it's not the default. It's a long story, @rajeee could explain in detail if you're interested.

@bonnema
Copy link
Contributor

bonnema commented May 3, 2021

Your explanation makes sense, just curious on my end 😀. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants