deploy container storage

This guide will review how to deploy different kinds of static registries. We currently have two kinds:

• Basic Registry the first is a basic example that builds and hosts the registry under one repository.
• Org Registry The second is an organizational registry that has a central repository for manifests that is updated via container builds from other repositories.
• Interaction how to interact with your registry manifest and tags endpoints.

The current examples all use CircleCI artifact storage for (non production) storage, and we will soon have examples that use Google Cloud Storage.

---

Basic

The basic registry is the registry deployed at the repository singularityhub/registry that makes the assumption that you want to store all of your build recipes within one repository, use CircleCI to build and deploy (saving images as artifacts).

1. Setup

To deploy this registry you can:

• Fork or download singularityhub/registry to your own GitHub account. The name that you choose will be the deployment URL for your registry (e.g., https://.github.io/
• Customize metadata such as the registry name and maintainer in the _config.yml
• Connect your repository to CircleCI
• Create a Github Machine User and make sure to add its SSH deploy key in the CircleCI settings.

2. Add a New Container

How does building work? Each folder represents a namespace. For example, the folder vanessa/greeting maps to a container collection that will be served at https://singularityhub.github.io/vanessa/greeting, with manifests for tags under https://singularityhub.github.io/vanessa/greeting/manifests/<tag>. This means that to add a new container collection namespace, just create a folder for it. Since we will store recipes organized by manifest, just make that folder path:

          # <container>/manifests/<tag>
$ mkdir -p vanessa/salad/manifests/latest

And since we store the recipe for each tag in the manifests folder, you can add the Singularity file there:

$ touch vanessa/salad/manifests/latest/Singularity

That's it! Write your recipe there, and then open a pull request to build the container. Once the container is built, you need to approve the Hold in the continuous integration, and then the rest of the container's metadata files will be generated and pushed back to the master branch.

Update a Container

I found it unreliable to rely on changes to Singularity files based on commits, so instead we take a simple approach. After you deploy a container, the continuous integration generates a README.md in the container's manifest tag folder. For example:

$ ls vanessa/salad/manifests/latest
   Singularity     # we generated this one
   README.md       # this was generated for us

To rebuild the container, simply delete the README.md in a pull request. It will be built and generated again to update the container.

---

Organizational

The org registry is a modified basic registry, deployed at https://singularityhub.github.io/registry-org/ with the change that the registry repository itself doesn't do any building. Specifically:

• other repositories each serve and build container recipes
• a pull request runs a continuous integration workflow to build and upload the container to storage
• when the deployment is done, the workflow finishes by creating a new branch (named according to its namespace, so each repository always has a unique branch)
• the creation or update of the branch opens a new pull request on the repository via a GitHub Action.

The registry would then do any testing desired to ensure that the objects exist, content types are valid, etc. I'll add this testing when I have a more solid example with several build options.

1. Setup

To deploy this registry you can:

• Fork or download singularityhub/registry-org to your own GitHub account. The name that you choose will be the deployment URL for your registry (e.g., https://.github.io/
• Customize metadata such as the registry name and maintainer in the _config.yml
• Create separate container repositories (instructions to follow)

2. Creating a Connected Repository

I've tested creating repositories within the same GitHub organization, but logically you could have other user names as long as the user submitting the pull request has write access to the organization repository. Here are steps to set up your container build repository:

1. Start with a template, such as singularityhub/centos. In the future we will have templates for different build and deploy settings. For now, know that the driving scripts are in the .circleci folder, and we are going to build a Singularity container and use CircleCI artifact storage.
2. Connect your repository to CircleCI
3. Create a Github Machine User and make sure to add its SSH deploy key in the CircleCI settings.
4. Proceed as you would to add containers (Singularity recipe files) in your repository.

Unlike the basic registry, the organizational registry has a namespace based on the GitHub repositories that submit containers to it! So if you are submitting a container from singularityhub/ubuntu, the container will be served from https://singularityhub.github.io/singularityhub/ubuntu, with manifests and tags under https://singularityhub.github.io/singularityhub/ubuntu/manifests/<tag>.

How do I create a tag for a container collection?

Unlike the basic registry, the tags for your containers follow the Singularity Hub standard. The extension of the Singularity file maps to a tag. For example, Singularity will build the tag "latest" and Singularity.pancakes will build the tag pancakes.

The rules for interaction with the API to get a container or tag manifest are the same for both kinds of registries.

Interaction

Get a Container

It's not clear to me yet how this will (or could) be integrated with a tool like Singularity or Docker, but you can easily get the url for the container (to wget) via a curl:

$ curl https://singularityhub.github.io/registry/vanessa/greeting/manifests/latest/ | jq --raw-output '.layers [] .urls [0]'
https://storage.googleapis.com/singularityhub/singularityhub/github.com/vsoch/singularity-images/130504089d5b2b44e2788992d0de75b625da6796/a1025471b564766d08bdf2cb062c795c/a1025471b564766d08bdf2cb062c795c.simg

And here is a different tag:

$ curl https://singularityhub.github.io/registry/vanessa/greeting/manifests/asciiart/ | jq --raw-output '.layers [] .urls [0]'
https://45-167740989-gh.circle-artifacts.com/0/singularity-containers/vanessa/greeting/asciiart/c64213394d9a8cd90cea402d8bf6bc9fdd98a2a435b6c3796f6f63cceda9b9c3.sif

If you are astute, you'll notice that the storage locations are different! This will be the next addition / example to add, how to specify different kinds of builders.