Supporting SLURM env vars for launching MAPDL configuration #2754

germa89 · 2024-02-08T14:27:32Z

As the title.

The idea is that on SLURM HPC clusters, PyMAPDL will read the SLURM job through the env vars that the SLURM manager creates, so it can launch MAPDL with the appropriate number of cores.

You can see in the code:

options = [
            # 4,  # Fall back option
            SLURM_CPUS_PER_TASK * SLURM_NTASKS,  # (CPUs)
            SLURM_NPROCS,  # (CPUs)
            # SLURM_NTASKS,  # (tasks) Not necessary the number of CPUs,
            # SLURM_NNODES * SLURM_TASKS_PER_NODE * SLURM_CPUS_PER_TASK,  # (CPUs)
            SLURM_CPUS_ON_NODE * SLURM_NNODES,  # (cpus)
        ]
nproc = max(options)

which is the way I use to decide how many cores should MAPDL instance use.

Same with memory:

        if SLURM_MEM_PER_NODE:
            # RAM argument is in MB, so we need to convert
            if SLURM_MEM_PER_NODE[-1] == "T":  # tera
                ram = int(SLURM_MEM_PER_NODE[:-1]) * 10**6
            elif SLURM_MEM_PER_NODE[-1] == "G":  # giga
                ram = int(SLURM_MEM_PER_NODE[:-1]) * 10**3
            elif SLURM_MEM_PER_NODE[-1] == "G":  # mega
                ram = int(SLURM_MEM_PER_NODE[:-1]) * 10**0
            elif SLURM_MEM_PER_NODE[-1].upper() == "k":  # mega
                ram = int(SLURM_MEM_PER_NODE[:-1]) * 10 ** (-3)
            else:
                ram = int(SLURM_MEM_PER_NODE)

I do not use the MAPDL -machines argument to specify the machines used. I don't think it is needed/compatible.

Adding memory option

ansys-reviewer-bot · 2024-02-08T14:27:40Z

Thanks for opening a Pull Request. If you want to perform a review write a comment saying:

@ansys-reviewer-bot review

for more information, see https://pre-commit.ci

…sys/pymapdl into feat/supporting-slurm-manager

src/ansys/mapdl/core/launcher.py

Co-authored-by: Maxime Rey <87315832+MaxJPRey@users.noreply.github.com>

koubaa · 2024-03-08T13:31:53Z

cc @greschd . This might be related to your interest in launcher configuration

wiz-inc-572fc38784 · 2024-03-14T09:32:01Z

Wiz Scan Summary


IaC Misconfigurations
Vulnerabilities
Sensitive Data
Total
Secrets

codecov-commenter · 2024-04-02T10:33:05Z

Codecov Report

Attention: Patch coverage is 74.72527% with 23 lines in your changes are missing coverage. Please review.

Project coverage is 84.07%. Comparing base (598e2e8) to head (84d4bc7).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2754      +/-   ##
==========================================
- Coverage   86.64%   84.07%   -2.57%     
==========================================
  Files          52       52              
  Lines        9553     9632      +79     
==========================================
- Hits         8277     8098     -179     
- Misses       1276     1534     +258

Adding typing

PipKat · 2024-05-15T20:18:19Z

doc/source/user_guide/mapdl.rst

@@ -1165,10 +1165,13 @@ These are described in the following table:
 |                                       |    export PYMAPDL_MAPDL_VERSION=22.2                                |
 |                                       |                                                                     |
 +---------------------------------------+---------------------------------------------------------------------+
+| :envvar:`PYMAPDL_ON_SLURM`            | With this environment variable set to ``FALSE``, you can avoid      |
+|                                       | PyMAPDL to detect that it is running on a SLURM HPC cluster.        |


Suggested change

| | PyMAPDL to detect that it is running on a SLURM HPC cluster. |

| | PyMAPDL from detecting that it is running on a SLURM HPC cluster. |

Sorry for messing up your table formatting with this needed correction!

PipKat · 2024-05-15T20:20:51Z

doc/source/user_guide/mapdl.rst

 |                                       | PRNSOL or NLIST, raise this. In bytes,                              |
 |                                       | defaults to 256 MB.                                                 |


PipKat · 2024-05-15T20:21:54Z

doc/source/user_guide/mapdl.rst

@@ -1165,10 +1165,13 @@ These are described in the following table:
 |                                       |    export PYMAPDL_MAPDL_VERSION=22.2                                |
 |                                       |                                                                     |
 +---------------------------------------+---------------------------------------------------------------------+
+| :envvar:`PYMAPDL_ON_SLURM`            | With this environment variable set to ``FALSE``, you can avoid      |
+|                                       | PyMAPDL to detect that it is running on a SLURM HPC cluster.        |
+---------------------------------------+---------------------------------------------------------------------+
 | :envvar:`PYMAPDL_MAX_MESSAGE_LENGTH`  | Maximum gRPC message length. If your                                |
 |                                       | connection terminates when running                                  |
 |                                       | PRNSOL or NLIST, raise this. In bytes,                              |
 |                                       | defaults to 256 MB.                                                 |
 |                                       |                                                                     |
 |                                       | Only for developing purposes.                                       |


This should be a complete sentence.

PipKat · 2024-05-15T20:22:49Z

src/ansys/mapdl/core/launcher.py

+        # Unless the env var is false, it will be true.
+        ON_SLURM = not (ON_SLURM.lower() == "false")
+
+    # Let's require the following env vars to exists to go into slurm mode.


Suggested change

# Let's require the following env vars to exists to go into slurm mode.

# Let's require the following env vars to exist to go into slurm mode.

PipKat · 2024-05-15T20:25:40Z

src/ansys/mapdl/core/launcher.py

+        ON_SLURM = True  # Using this as main variable
+    else:
+        ON_SLURM = False
+
    if remove_temp_files is not None:
        warnings.warn(
            "The option ``remove_temp_files`` is being deprecated and it will be removed by PyMAPDL version 0.66.0.\n"


Suggested change

"The option ``remove_temp_files`` is being deprecated and it will be removed by PyMAPDL version 0.66.0.\n"

"The ``remove_temp_files`` option is being deprecated. It is to be removed in PyMAPDL version 0.66.0.\n"

I'm not sure if you were trying to avoid being specific by saying "by PyMAPDL version 0.66.0" because it could be before then? Perhaps "It is to be removed sometime before PyMAPDL version 0.66.0."

PipKat · 2024-05-15T20:26:13Z

src/ansys/mapdl/core/launcher.py

-            raise NotEnoughResources
+            if machine_cores < int(nproc):
+                raise NotEnoughResources(
+                    f"The machine has {machine_cores} cores and PyMAPDL is asking for {nproc} cores."


Suggested change

f"The machine has {machine_cores} cores and PyMAPDL is asking for {nproc} cores."

f"The machine has {machine_cores} cores. PyMAPDL is asking for {nproc} cores."

PipKat · 2024-05-15T20:26:58Z

src/ansys/mapdl/core/launcher.py

+
+        except Exception as e:
+            LOG.info(
+                f"The machines list could not be obtained.\nFollowing error happened:\n{str(e)}"


Suggested change

f"The machines list could not be obtained.\nFollowing error happened:\n{str(e)}"

f"The machines list could not be obtained.\nThis error occurred:\n{str(e)}"

PipKat · 2024-05-15T20:27:36Z

tests/conftest.py

@@ -455,6 +455,31 @@ def run_before_and_after_tests_2(request, mapdl):
    assert prev == mapdl.is_local


+@pytest.fixture(scope="function")
+def set_env_var(request, monkeypatch):
+    """Set an environment variable from given requests, this fixture must be used with `parametrize`"""


Suggested change

"""Set an environment variable from given requests, this fixture must be used with `parametrize`"""

"""Set an environment variable from given requests. This fixture must be used with ``parametrize``."""

PipKat · 2024-05-15T20:28:07Z

tests/conftest.py

+
+@pytest.fixture(scope="function")
+def set_env_var_context(request, monkeypatch):
+    """Set MY_VARIABLE environment variable, this fixture must be used with `parametrize`"""


Suggested change

"""Set MY_VARIABLE environment variable, this fixture must be used with `parametrize`"""

"""Set ``MY_VARIABLE`` environment variable. This fixture must be used with ``parametrize``."""

PipKat · 2024-05-15T20:28:45Z

tests/test_launcher.py

+            id="Testing NPROCS only",
+        ),
+        pytest.param(
+            # This test does not do probably a good memory mapping between


Suggested change

# This test does not do probably a good memory mapping between

# This test probably does not do a good memory mapping between

German Martinez Ayuso and others added 11 commits October 27, 2023 18:49

Adapting launcher to run on slurm

45b9bd3

Working on nodes parser

21089aa

Implemented machine argument

392cf5e

Update

70cd4a4

Adding more debugging

25b74dd

Adding memory option

Merge branch 'main' into feat/supporting-slurm-manager

86a0bf2

Avoiding checking number of processors in slrum

6e206a1

Merge branch 'main' into feat/supporting-slurm-manager

2ad509e

Cleaning empty args

157a941

renaming argument

ac7dc04

Update env vars to check to decide if ON_SLURM or not.

e95ed69

germa89 added the New Feature Request or proposal for a new feature label Feb 8, 2024

germa89 requested review from smedikon, clatapie and MaxJPRey February 8, 2024 14:27

germa89 self-assigned this Feb 8, 2024

github-actions bot added Documentation Documentation related (improving, adding, etc) Enhancement Improve any current implemented feature labels Feb 8, 2024

pre-commit-ci bot and others added 7 commits February 8, 2024 14:28

[pre-commit.ci] auto fixes from pre-commit.com hooks

d75b360

for more information, see https://pre-commit.ci

Removing unneeded

0266018

Merge branch 'feat/supporting-slurm-manager' of https://github.com/an…

556b528

…sys/pymapdl into feat/supporting-slurm-manager

Adding comment

fb43dfd

removing redundancies

657637a

Removing self

1b1f17e

Removing unused arg

63ee786

MaxJPRey reviewed Feb 9, 2024

View reviewed changes

src/ansys/mapdl/core/launcher.py Outdated Show resolved Hide resolved

MaxJPRey reviewed Feb 9, 2024

View reviewed changes

src/ansys/mapdl/core/launcher.py Outdated Show resolved Hide resolved

germa89 and others added 2 commits February 9, 2024 15:55

Update src/ansys/mapdl/core/launcher.py

574542a

Co-authored-by: Maxime Rey <87315832+MaxJPRey@users.noreply.github.com>

fixing memory units

77ae2ab

germa89 added 4 commits March 7, 2024 17:54

Adding tests

873bdb5

Moving fixtures to main file

b599c37

Merge branch 'main' into feat/supporting-slurm-manager

e40782c

testing exec_file

88c8ea9

germa89 mentioned this pull request Mar 8, 2024

PyMAPDL and HPC clusters #2865

Open

Merge branch 'main' into feat/supporting-slurm-manager

7033bd4

german and others added 3 commits March 18, 2024 16:45

Merge branch 'main' into feat/supporting-slurm-manager

29a82d6

Merge branch 'main' into feat/supporting-slurm-manager

f58dd22

Merge branch 'main' into feat/supporting-slurm-manager

bd9f380

german added 4 commits April 2, 2024 11:02

fix tests

f777c76

Adding env var documentation

20fb11a

Small refactoring regarding env var processing

1824ede

Adding typing

Better env var order

4d1cd60

germa89 mentioned this pull request Apr 4, 2024

HPC documentation #2966

Merged

Merge branch 'main' into feat/supporting-slurm-manager

84d4bc7

germa89 marked this pull request as ready for review May 13, 2024 16:24

germa89 marked this pull request as draft May 13, 2024 16:25

PipKat reviewed May 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Supporting SLURM env vars for launching MAPDL configuration #2754

Supporting SLURM env vars for launching MAPDL configuration #2754

germa89 commented Feb 8, 2024 •

edited

ansys-reviewer-bot bot commented Feb 8, 2024

koubaa commented Mar 8, 2024

wiz-inc-572fc38784 bot commented Mar 14, 2024 •

edited

codecov-commenter commented Apr 2, 2024 •

edited

PipKat May 15, 2024 •

edited

PipKat May 15, 2024

PipKat May 15, 2024

PipKat May 15, 2024

PipKat May 15, 2024

PipKat May 15, 2024

PipKat May 15, 2024

PipKat May 15, 2024

PipKat May 15, 2024

PipKat May 15, 2024

	\| \| PyMAPDL to detect that it is running on a SLURM HPC cluster. \|
	\| \| PyMAPDL from detecting that it is running on a SLURM HPC cluster. \|

		\| \| PRNSOL or NLIST, raise this. In bytes, \|
		\| \| defaults to 256 MB. \|

	# Let's require the following env vars to exists to go into slurm mode.
	# Let's require the following env vars to exist to go into slurm mode.

	"The option ``remove_temp_files`` is being deprecated and it will be removed by PyMAPDL version 0.66.0.\n"
	"The ``remove_temp_files`` option is being deprecated. It is to be removed in PyMAPDL version 0.66.0.\n"

	f"The machine has {machine_cores} cores and PyMAPDL is asking for {nproc} cores."
	f"The machine has {machine_cores} cores. PyMAPDL is asking for {nproc} cores."

	f"The machines list could not be obtained.\nFollowing error happened:\n{str(e)}"
	f"The machines list could not be obtained.\nThis error occurred:\n{str(e)}"

	"""Set an environment variable from given requests, this fixture must be used with `parametrize`"""
	"""Set an environment variable from given requests. This fixture must be used with ``parametrize``."""

	"""Set MY_VARIABLE environment variable, this fixture must be used with `parametrize`"""
	"""Set ``MY_VARIABLE`` environment variable. This fixture must be used with ``parametrize``."""

	# This test does not do probably a good memory mapping between
	# This test probably does not do a good memory mapping between

Supporting SLURM env vars for launching MAPDL configuration #2754

Are you sure you want to change the base?

Supporting SLURM env vars for launching MAPDL configuration #2754

Conversation

germa89 commented Feb 8, 2024 • edited

ansys-reviewer-bot bot commented Feb 8, 2024

koubaa commented Mar 8, 2024

wiz-inc-572fc38784 bot commented Mar 14, 2024 • edited

Wiz Scan Summary

codecov-commenter commented Apr 2, 2024 • edited

Codecov Report

PipKat May 15, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

germa89 commented Feb 8, 2024 •

edited

wiz-inc-572fc38784 bot commented Mar 14, 2024 •

edited

codecov-commenter commented Apr 2, 2024 •

edited

PipKat May 15, 2024 •

edited