Integrating JUDI with Cloud Services for Distributed Computing

[NikoYura]

I'm working on setting up a distributed computing environment for a JUDI. I have several questions about integrating JUDI with cloud services. I would gratefully appreciate insights and suggestions from the community on the following aspects:

Setting Up Distributed Computing in Cloud with JUDI:

How can I effectively connect JUDI with distributed computing capabilities of a Cloud setup (shot parallelization over many nodes)?
My overall task is to somehow connect workers from FWI JUDI examples with my nodes. I did the following:

• I checked the example with implementing Azure services. As I understand, to implement these packages (AzManagers, AzSessions, AzStorage) for any other cloud, I will need to rewrite the code inside these packages to match my specific cloud.

• I used addprocs() with SSH, IP settings to connect to the node. It works but you need the copy of your environment on all other nodes. So it is hard to ensure consistent JUDI environments across different nodes in Cloud. What strategies are recommended for maintaining consistent package versions across all nodes in a JUDI distributed computing environment? Is it possible to manage this seamlessly using tools like Dask or Kubernetes, and how does this translate to a JUDI context?

• Maybe some manager for addprocs is best for this purpose (shot parallelization over many nodes)? For example Slurm? How labor intensive is it to configure this? What are the key considerations for setting up and customizing Slurm job parameters in JUDI?

• I also followed this tutorial but for my specific cloud service. I took Devito image from Docker Hub and configured Kubernetes. It works for the FWI Devito example with DASK. But I do not understand how to make the same configuration/code/setup for JUDI (FWI examples with parallelization based on workers).

I realize that my questions overlap in general, but I have only one task - to parallelize shots across nodes (and as I understand it, the easiest way to do it is via wokers in the FWI JUDI examples). At this stage, I'm not sure which way would be best (at least easy to implement), and would welcome your perspective on this issue. I'm aiming to build an efficient, scalable, and robust distributed computing setup for my FWI application and would greatly benefit from the community's expertise in these areas.

[kerim371]

@NikoYura hi,

From may experience of working with JUDI on the cloud the main aspects are:

1. it is convenient to have one master node and as much as you need computing nodes
2. master node should have shared folder and julia, python, compiler should be located there. Also JL_DEPOT should be shared folder
3. each computing node should have only consistent environment probably setup in ~/.bashrc with vars like: PYTHON, JULIA, CC, CXX, LD_LIBRARY_PATH (to the compiler lib folder and python lib dir), adn PATH that contains python/julia bin dirs
4. it is better to have master (and probably computing nodes) with Linux distribution that comes with GCC >= 9 or you may experience troubles with OpenMP parallelization. Another way is to build GCC from source and locate it in shared dir (on CentOS 7 for example)

Julia should be run from shared folder as I remember.
SSH cluster manager works pretty good.
I tried to setup SLURM on Centos7 and succeded in installation but at some point when cluster was working I used to get desinchronization error or something like that. I don't know the reason of that but that made me using SSH manager instead.
Ansible helps to setup computing nodes.

[NikoYura]

@kerim371 Hi!

Thank you for your answers!

If I understand correctly, you have Slurm Manager working well in conjunction with the Distributed package (Julia) and the addprocs command? How did you use them together?

And in general, is it necessary to mess with the environment on controlled nodes or is it easier to work with containerization and send ready containers (images) to nodes and manage them (Kubernetes or something else?)?

Also, @mloubout, please, do you have any comments regarding my issue?

[mloubout]

Hi @NikoYura thanks for reaching out and glad you are using JUDI

Which cloud provider are you interested in? For Azure, there is a little wrapper for Azure Batch that is very easy to use you can look at

https://github.com/slimgroup/JUDI4Cloud.jl

I find it easier to use than the AzManager and such (but it does use AzStorage under the hood if needed as well)

In general, you will need to make sure your environment is consistent for all workers as julia tends to be very gimmicky when there is a package version mismatch (and will usually completely crash if there is Julia version mismatch). The simplest way to ensure this is to set up your project with a Manifest.toml included and always run instantiate first to enforce package versions. You can have a look at DrWatson.jl which offers a very nice workflow for reproducibility and stable environment.

[mloubout]

Yes it would these images are automatically updated at each new version of JUDI

[NikoYura]

Thank you for your anwers!

Sorry for the unrelated message, but I don't wanna start a new issue. I noticed that if I generate observed data using any generator in JUDI and deliberately violating CFL condition (let's say my cmax*dt/dx > 1/sqrt(2) for 2D case), it still generates seismograms. Do you have some kind of internal dt check and correction for stable calculation?

[mloubout]

There is two different time axis, the seismogram one that you specify when you create a geometry, and the internal computational time step that is computed based on the velocity at runtime and JUDI interpolates between the two for propagation.

If you want to specify the computational time step you can set the dt_comp option in Options but if not it will use the CFL based one and return the seismogram on the specified data dt

[NikoYura]

Ah, thank you so much for that clarification! Now all the logic is clear.

Is there any possibility to add an absorbing layer, as it is done in devito by setting the nbl parameter? I couldn't find it in the examples.

[mloubout]

There is always an absorbing layer with a default width of 40 points which can be modified by providing the nb arguemtn to Model, i.e Model(n, d, o, m; nb=80) to make it 80 points wide