Confusion about global versus local arrays

[jklymak]

If I run in parallel mode (eg mpirun -np 4 veros run --force-overwrite example.py -n 4 1) when I drop into a routine like set_grid the variables I want to set are no longer global. eg. if settings.nx=200, then vs.dxt has x dimension 54 (200/4 + 4). If I want to set this in a way that varies along x (stretched grid) I don't see a way to know what tile I'm currently on, or what the global index is. Am I missing a variable?

[jklymak]

Can I just use distributed.scatter for this?

[dionhaefner]

The easiest way to achieve this within setup files is to pass dist_safe=False to @veros_routineand to work on global arrays instead. Routines marked as non-dist safe will automatically perform gather and scatter operations at their boundaries. The code is executed on the main process which shouldn't be an issue for routines that are only called once (i.e., all except set_forcing and after_timestep).

Otherwise, you can use veros.runtime_state.proc_idx to get the (i, j) index of the current process.

[jklymak]

Thanks! Somehow Distributed Memory didn't twig it for me.

I do wonder if some of the cheaper init routines should default to this, particularly set_grid.

[dionhaefner]

That's certainly an option. There's one more footgun involved if the global arrays are too large to fit into memory though. (Not a huge concern for set_grid specifically.)