Environment variable with intel MPI

[mloubout]

Problem Description

Some environment variables are hard-coded for intel mpi (on supported VMs HC and HB). I ran in this issue running MPI jobs on 2 nodes with intel mpi 2020, where mlx does not seem to work but verbs does for FI_PROVIDERS but I had to go change it in batch.py to make it work.

Batch Shipyard Version

Latest

Steps to Reproduce

Set FI_PROVIDERS or I_MPI_FABRICS as an environment variable in the jobs.yaml

Expected Results

Set env variables to what is defined in jobs.yaml

Actual Results

Defaults from batch.py line 4409 and 4410

Redacted Configuration

INSERT RELEVANT YAML FILES

Additional Logs

INSERT ADDITIONAL LOGS HERE

Additonal Comments

Haven't find where to get it but these liens (and maybe for other versions too) may be better replace by an `ib_env[...] = env.get(user) or default (pseudo-code)

[alfpark]

Intel MPI 2020 is untested/unverified.

Do you have the Mellanox OFED stack installed on your image/VM?

[mloubout]

Yes I do have MLNX_OFED installed from Mellanox. I got it to run ith some UCX fallback variables, but I think it would be nice to be able to set these environment variables differently rather than having to debug the hardcoded shm:ofi+mlx