HuggingFace Refactor

[ludgerpaehler]

• Pushing geometries, checkpoints, and checkpoint out into dedicated HuggingFace data repositories.
  • HydroGym Checkpoints
  • HydroGym Meshes

[jcallaham]

Great. Could you maybe add an example or note in the docs about how users can pull this data? Maybe people will be familiar with HF repos, but at least I didn't know about them.

And let's actually keep the .geo files in the GH repo if that's okay with you. These are small and human readable, and it'd be nice to be able to see changes to mesh definitions easily. The other data is convenient but optional, since someone could rebuild it all themselves.

[ludgerpaehler]

Tasks to completion:

• Move geometry files back in
• Auto-call to the Huggingface-API doing away with the os-calls
• Add respective subpage to the developer's documentation

[ludgerpaehler]

Great. Could you maybe add an example or note in the docs about how users can pull this data? Maybe people will be familiar with HF repos, but at least I didn't know about them.

The data-loading will be automatic, hidden away from the user.

[ludgerpaehler]

And let's actually keep the .geo files in the GH repo if that's okay with you.

I'd rather not. Leaving them out in the dataset repo allows us to purge the os-dependency, which we shouldn't have. I'd rather load them directly through the dataset API..

[ludgerpaehler]

Or we need to find a solution with the .geom files which does not involve os-calls. I see the utility of having them in the main tree to keep tabs, but we really shouldn't have os-calls. It is really bad practice and might easily fail users on clusters.

[ludgerpaehler]

@jcallaham same is valid for the default configurations btw, will try to move them out to their own json configurations later today s.t. they are isolated and we don't carry values in the configuration code anymore.

[ludgerpaehler]

WIP not merge-ready

[jcallaham]

Or we need to find a solution with the .geom files which does not involve os-calls. I see the utility of having them in the main tree to keep tabs, but we really shouldn't have os-calls. It is really bad practice and might easily fail users on clusters.

I'm good with that - when the built-in meshes can be pulled from HF we can get rid of the convenience of not having to specify the full path manually, which is all the os call does. Then we can move the .geo files to a meshes/ directory or something. But changes to the meshes need to be tracked in git, and there still needs to be a way of pointing to a local .msh file if you want to modify the geometry or create a custom environment. The HuggingFace repo should be a convenience but not a hard requirement.

Also if I wanted to rename the meshes away from the "coarse", "medium", "fine" towards something a little less misleading, is it too late to do that?