LMOD: Error while loading shared libraries: libpython3.10.so.1.0: cannot open shared object file: No such file or directory

[BramVanroy]

At our cluster, we us lmod for modular module usage. So when starting a new job, we need to load existing modules, e.g.:

To load PyTorch, Python, and pdsh:

module load PyTorch/1.12.0-foss-2022a-CUDA-11.7.0
module load pdsh/2.34-GCCcore-11.3.0

So if I start a multi-node, interactive job to test deepspeed, I have these activated on my main node. When I then run some deepspeed code, I get an error that Python cannot be found.

gpu510: Warning: Permanently added 'gpu510.dodrio.os,IP-ADDRESS' (ECDSA) to the list of known hosts.
gpu509: mesg: ttyname failed: Inappropriate ioctl for device
gpu510: mesg: ttyname failed: Inappropriate ioctl for device
gpu509: /dodrio/scratch/projects/someproject/deepspeed_test/.venv/bin/python: error while loading shared libraries: libpython3.10.so.1.0: cannot open shared object file: No such file or directory
pdsh@gpu509: gpu509: ssh exited with exit code 127
gpu510: /dodrio/scratch/projects/someproject/deepspeed_test/.venv/bin/python: error while loading shared libraries: libpython3.10.so.1.0: cannot open shared object file: No such file or directory
pdsh@gpu509: gpu510: ssh exited with exit code 127

I believe the reason is that pdsh ssh's into the nodes but of course not automatically loads the correct modules. So Python is not available to the nodes after ssh.

I am not sure how I can solve that. Is there a file of commands that I can give to deepspeed/pdsh that it needs to execute right after ssh'ing?

[tjruwase]

This is a tricky one. Can you try adding these commands into your .bashrc?

[BramVanroy]

That seems to work indeed, great idea! The downside is that this will try to load the modules at every session, which I do not want. Is there a variable that is set by either pdsh or deepspeed during the session that I can check against, so that the .bashrc only loads the modules when running deepspeed?

[AI-General]

which command do you mean?

[UlrikHjort]

Guess he means the load module commands above:
module load PyTorch/1.12.0-foss-2022a-CUDA-11.7.0
module load pdsh/2.34-GCCcore-11.3.0

[tjruwase]

Glad to hear of the progress. Yes, I understand the downside of the current approach. I don't there is any environment variable set on login by pdsh/deepspeed. However, you could test for this dumping the env vars in the first very line of your .bashrc.

Another, somewhat heavyweight option, is to create a deepspeed user in your cluster dedicated for deepspeed runs. In that case, you could specialize the .bashrc to always load the required modules.

[BramVanroy]

Thanks again for the swift reply. I did some digging, and I found that the best way to deal for us would be to check which kind of node we were running on. We have GPU and non-GPU nodes, so I did not want to load PyTorch+CUDA on the non-GPU nodes every time. Luckily our GPU-nodes' names all start with gpu. So I added the following to my .bashrc, which works well.

# If we open job that's on host that starts with gpu* (e.g. gpu512.dodrio.os), load PyTorch with CUDA
# This also makes sure that deepspeed/pdsh work in multi node settings
if [[ $(hostname) == gpu* ]]; then
    module load PyTorch/1.12.0-foss-2022a-CUDA-11.7.0
    module load pdsh/2.34-GCCcore-11.3.0
fi

More in this gist, with also a function to automatically generate the hostfile based on SLURM nodes + pdsh.

Thanks again for the advice!