Improvements on filemovements

[mandresm]

This PR aims to:

• Remove the intermediate copying by default (except for config, input, and forcing. The defaults can be overridden by general.intermediate_movement = [list of filetypes that go to the intermediate folders]
• [to be done in another PR] Parallelize the file movements (WIP)
• Profiling

This is to reduce the copying times of @tsemmler05, @a270067 and @mjy1011.

@JanStreffing and @mjy1011, this branch is not in it's final stage, but it would be a great help if you can check that is working for an actual simulation. I've tested already a bit but not in depth.

[JanStreffing]

This week I will be back. Hopefully be Wednesday or so, I get to give this a test.

[mandresm]

I consider this finished, so feel free to give it a try. I will work on the parallelization on another PR, and that would take longer.

PD: Wednesday is bank holiday ;)

[mjy1011]

Hi Miguel, thank you very much for the modification.
On Aleph when we restart experiment with the updated codes, attached error occurs.
It seems oifs restart info files (rcf, waminfo)' path are not detected correctly.

Do we need to modify any other files? for example, /esm_tools/configs/esm_software/esm_runscripts/defaults.yaml ?

[mandresm]

Hi @mjy1011, I'll be looking into this today. The problem is that the preprocessing scripts expect files in the intermediate folders.

[mandresm]

Hi @mjy1011,

I've got done some progress today with the issue, but there are some weird errors still. I think I am close, but I'll have to continue tomorrow.

[mjy1011]

Hi @mjy1011,

I've got done some progress today with the issue, but there are some weird errors still. I think I am close, but I'll have to continue tomorrow.

That's nice. Thank you for the update.

[mandresm]

Hi @mjy1011,

That was a real tough one... but I think it's working now. Can you please try?

@JanStreffing, can you try if the results are bit identical with release and this branch? I am getting non-bit identical results in some OpenIFS output files and restarts, and I am wondering if that's normal? (for example, maybe the results in the files are identical but there is something different in the headers)

[mjy1011]

Hi @mjy1011,

That was a real tough one... but I think it's working now. Can you please try?

@JanStreffing, can you try if the results are bit identical with release and this branch? I am getting non-bit identical results in some OpenIFS output files and restarts, and I am wondering if that's normal? (for example, maybe the results in the files are identical but there is something different in the headers)

Hi ~~
I am waiting for the pre-running job to be finished. I'll try and let you know in a few days. Nice work again!

[pgierz]

Looks good!

Only a few minor changes (and one typo 😉)

[mandresm]

Looks good!

Only a few minor changes (and one typo 😉)

Thanks for the review!

[pgierz]

I agree with the error approach!

[mjy1011]

Hi @mjy1011,
That was a real tough one... but I think it's working now. Can you please try?
@JanStreffing, can you try if the results are bit identical with release and this branch? I am getting non-bit identical results in some OpenIFS output files and restarts, and I am wondering if that's normal? (for example, maybe the results in the files are identical but there is something different in the headers)

Hi ~~ I am waiting for the pre-running job to be finished. I'll try and let you know in a few days. Nice work again!

Just confirmed it is working right! And most of all, saving more than 2 hours, amazing. Thank you very much!

[mandresm]

#approve-changes

[mandresm]

Just confirmed it is working right! And most of all, saving more than 2 hours, amazing. Thank you very much!

You are welcome, and thanks for your patience with this issue :)

[mandresm]

#bump