Skip to content

vsoch/containershare

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

65 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Containershare

This is an open source library of containers. You can come here for one of several use cases:

  1. You want to use a container that is available from the browsable library provided by the registry. You can use the container locally, or as a suggestion for a shared resource that requires a graphical forward, you can use the forward tool.
  2. You want to contribute or use a containershare template. Contributing means that your code gets packaged in a reproducible way, deployed for sharing, and with exposure of metadata like packages and files.
  3. You want to deploy your own open source containershare all with freely available, easy to use tools that you are already familiar with.

Read about the development and use cases here, and please reference the software if it is useful to you:

DOI DOI

Continue reading for details on the above, or ask a question if you need help. Or jump down to one of our templates:

Container Repository Templates

Here are some helpful templates to get you started! Detailed instructions and usage can be found in their respective repositories.

For all of the above, jump down to the links to try out examples! If you have a request for a new template, please ask.

Use Cases

Run a Container

You can browse available containers here. You can use a container is several ways!

  1. Follow instructions to configure the forward tool. There are also good instructions available in tutorials linked from that repository. Containers in the containershare library that are "forward friendly," meaning you are intended to use ssh forwarding from your local machine to interact with a web based notebook, are tagged with forward.
  2. Find a container you like, and pull directly with Docker or Singularity for a shared HPC resource.
  3. Use the containershare library (under development) that will allow you to search container metadata and inspected content.

Containershare Templates

Step 1. What kind of container?

Whether you want to contribute or use a container, the first step is to decide what kind of container you want. Below we have getting started guides and templates for multiple kinds, along with examples for you to quickly demo them. For all of the containers below, we name them repo2docker and use the same port (8888) so be sure to change this if you want to run multiple at the same time. Or just kill the running container and do a la carte.

Repo2Docker

Do you have a jupyter (or similar) notebook and want to quickly build and deploy it? You will want to use the repo2docker share template. This is based on the continuous builder.

docker run --rm -it -p 8888:8888 vanessa/repo2docker-share jupyter notebook --ip 0.0.0.0

Repo2Docker R

If you use R, this container comes with R, RStudio, and an example of Shiny! The repo2docker-r template will allow you to add a runtime.txt file, a notebook file, along with an install.R file to build the perfect container.

docker run --rm -it -p 8888:8888 vanessa/repo2docker-r jupyter notebook --ip 0.0.0.0

Repo2Docker Conda

Want to deploy a container with your favorite conda installs? This template will make it easy to install miniconda, and then run your notebook with dependencies ready to go! Want to get started? Check out the repo2docker-conda template.

docker run --rm -it -p 8888:8888 vanessa/repo2docker-conda jupyter notebook --ip 0.0.0.0

Repo2Docker Julia

This is a complete example of a Julia+Jupyter notebook, built with repo2docker for you to use! The repo2docker-julia template drives this with a simple environment.yml file for Python dependencies, and REQUIRE file for Julia. Another template for jupyter (that doesn't use repo2docker) is the julia-share

docker run --rm -it -p 8888:8888 vanessa/repo2docker-julia jupyter notebook --ip 0.0.0.0

General Docker

If you have a general Docker container that is built from a Dockerfile in your repository, check out the docker-share template.

docker run --rm -it -p 8888:8888 vanessa/repo2docker-share jupyter notebook --ip 0.0.0.0

Do you have another container template you'd like? Let me know! Another good way to start is to browse the table, find a container like yours, and then copy the circle configuration. If you are using the container, you can stop here. If you want to contribute a container template (perhaps starting from one of these bases) keep reading to learn how to set up the continuous build to generate the Docker image and metadata on Github Pages.

Step 2. Submit Metadata

After doing the above, you will have a repository that is hooked up to continuous integration, and deploys to Docker Hub. The rest from here is easy! You just need to submit this repository. How do you do that?

  1. Fork this repository, pull your fork on your local machine, and create a new branch for your container.
  2. Once the container is deployed on Docker Hub (step 1) add a markdown file to the _library folder.
  3. Open a pull request to the master branch to add your contribution!

When the pull request is merged, the container will be added to the table and library for others to use.

Deploy a Containershare

If you want to deploy your own containershare, there are two components that you can do.

  1. Is the web interface here
  2. Is a folder of shared containers on your research cluster that corresponds to them.

1. Deploy the Container Share Interface

You may just be interested in having the online library of tested containers, and so you would want to start here! For the interface, it's just a matter of forking this repository, and turning on Github pages to deploy from the docs folder, and then connecting the repository to circleci. Once this is done, third parties (others) should be able to equivalently submit pull requests to your registry that are tested, and upon merge, the container contribution added to the table, and the container available for discovery via the API exposed by the registry.

2. Deploy a containershare on your cluster

If you use the forward tool there is a script that will allow users to easily access a shared folder of containers that you provide. That's right... the container share! See the script create.py and the associated README to pull the containers. Then check out the forward tool's sherlock/containershare-jupyter.sbatch (and similar) scripts for your users to use your containers!

Note that the create.py will be replaced with a better solution, this was a quick script that @vsoch threw together!

Local Tests

The testing step of the registry that occurs on CircleCI is optimized to only test newly added files (so previous additions do not need to be tested and take, however the testing can be run locally (and manually) if desired. The general steps would be to do the following:

  • build the site locally with jekyll
  • inspect the image manifest
  • for each new container represented as a markdown file in the library, check that it serves metadata

Step 1. Build the Jekyll Site

You can install jekyll for your platform of interest], and build the site after cloning the repository:

git clone https://www.github.com/vsoch/containershare
cd containershare/docs
jekyll build

This will generate static content for the site (including the library.json API file) in the _site folder. Check it out!

cat _site/library.json

Step 2. Install Containershare

Containershare has a small python library that is used to run the tests, and this is done to ensure versioning of the testing itself. You can install this library (check the .circleci/config.yml for the updated version, at the time of this writing the current version is 0.0.14) and then use it to run tests on the static generated content.

pip install pyaml containershare==0.0.14

It's assumed that you have some flavor of python installed, along with the package manager pip. To use the exact version that the CircleCI is using, these are the steps to install:

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
/bin/bash Miniconda3-latest-Linux-x86_64.sh -b 
$HOME/miniconda3/bin/python -m pip install pyaml containershare==0.0.14

Step 3. Test the library

The final command will run the tests! If you are in the "docs" folder, go up one level back to the base of the repository ("tests" should be in the present working directory). Note that in the CircleCI continuous integration, the tests are run like this:

python -m unittest tests.test_library

The above command will run through the library, and for each entry, clone the Github repository Github pages and master branches, and check for all metadata files (tags.json, inspections and image manifests) along with "human friendly" metadata like a LICENSE and README. Finally, the tests ensure that the Github Pages (or other preview link) is deployed based on a 200 status return, and that the name does not include any invalid characters (outside of numbers, letters, - _ and /).