Maintaining a Singularity image off a docker container

[smokhov]

Hello,

I plan on building Singularity images for our HPC cluster. There are some that we will maintain on Docker hub or some that are maintained by others. I plan producing Singularity recipes that fetch these docker images and build corresponding Singularity containers. Is this tool docker2singularity a proper way to do it? E.g., if I updated and maintain https://hub.docker.com/r/openiss/rt-mocap-vfx and then create a Singularity file that transforms to the corresponding container here?

• Possibly related to Using docker2singularity to build images from Singularity definition files #98
• The example here uses Bootstrap:docker, not sure if it's relevant

Thank you!

[vsoch]

hey @smokhov ! Your best bet is probably to just pull to Singularity directly from Docker Hub, e.g.,:

singularity pull docker://openiss/rt-mocap-vfx 

so you don't have to manage a bunch of separate Singularity recipes. If you decided to use docker2singularity, you'd need to have a place (probably off of your cluster) with Docker installed where you would do the conversion to generate the images (one step and one dependency more, which doesn't make sense unless you have a good reason to do it). Docker2singularity was developed before this pull functionality was working in full, so its not (in my mind) the preferred method anymore. But it does have a small advantage of being able to set different environment / metadata, or a working directory, if you absolutely need that.

The boostrap: docker will build a Singularity image with a docker base, but you might as well put your entire image in a Docker container and pull down to singularity on your resource (does not require sudo permissions).

TLDR: start with singularity pull docker:// and only explore docker2singularity if you need further customization, and cannot do it on the level of the image builds.

[smokhov]

hey @smokhov !

Thanks for the quick reply and thanks for your work on all the Singularity-related projects:

Your best bet is probably to just pull to Singularity directly from Docker Hub, e.g.,:

singularity pull docker://openiss/rt-mocap-vfx 

I am aware of that and tested that already. There are to issues with that:

• each pull each time will have build the singularity container first; I'd rather make images available by default having the users just have the images available. So as to save a step.
• I'd like other users who preferentially use Singularity elsewhere to find my containers in case they find them useful via the Singularity eco-system without necessarily the dockerhub being there.

so you don't have to manage a bunch of separate Singularity recipes.

I may have to build separate a few for several repos to prepackage and to make them available to our HPC community.
I have more than one container to build and to upload. So I wanted to have for each collection to push to GitHub and then both Docker and Singularity images are made, for example.

If you decided to use docker2singularity, you'd need to have a place (probably off of your cluster)
with Docker installed where you would do the conversion to generate the images
(one step and one dependency more, which doesn't make sense unless you have a good reason to do it).

I guess my question was more "if this is intended way" to use or if the purpose of this project
is different. I also thought #98 was proposing something of this sort.

Docker2singularity was developed before this pull functionality was working in full,
so its not (in my mind) the preferred method anymore. But it does have a small
advantage of being able to set different environment / metadata, or a working directory,
if you absolutely need that.

Thank you for clarifying that! I am still exploring, and I will keep that note on that "small advantage" you mention in case I need it.

The boostrap: docker will build a Singularity image with a docker base, but you
might as well put your entire image in a Docker container and pull down to singularity
on your resource (does not require sudo permissions).

TLDR: start with singularity pull docker:// and only explore docker2singularity if you need
further customization, and cannot do it on the level of the image builds.

Thanks!

[vsoch]

each pull each time will have build the singularity container first; I'd rather make images available by default having the users just have the images available. So as to save a step.

if you are using a non-old version of Singularity (not 2.x) the cache should reliably be used to grab the image instead of pulling again. This is on a per user basis, so if you wanted users to share you could 1. try using a shared cache, or 2. provide images as executables via some other method like modules.

I'd like other users who preferentially use Singularity elsewhere to find my containers in case they find them useful via the Singularity eco-system without necessarily the dockerhub being there.

At the end of the day you need a registry. You can either provide the images for your users, and always need to build / allocate when a new one is needed, or empower users to bring their own, either building locally and uploading to your cluster or (what I think is easiest) pulling from an OCI registry.

I guess my question was more "if this is intended way" to use or if the purpose of this project
is different. I also thought #98 was proposing something of this sort.

This would definitely be a use case, I didn't mean to downplay it! I would talk to the author of the issue and see if you can help to create an example / help with his. Maybe you could work together to help for your use case as well!