Technologies
This article covers the need-to-know technologies used by views. Maintainers should be reasonably familiar with the following:
Docker is the core technology to grasp for maintainers of ViEWS 3. All ViEWS 3 services use Docker to facilitate predictable deployments, providing a 1:1 dev-prod equivalence with complete portability. For ViEWS 3, Docker provides a simple, predictable and portable deployment experience.
For administrators, Docker is the tool used to spin up ViEWS 3 on a server, either locally for development, or on a production server. It is important to understand Docker image management through Docker registries, and to understand how to manage running Docker containers for monitoring.
For maintainers, it is important to be quite familiar with docker. Maintainers should understand how to write Dockerfiles and build Docker images.
While Docker can be a bit difficult to understand at first, it will become your favorite tool once you've become used to it! As an added bonus, Docker is the best (and only?) way to ensure complete reproducibility of computational processes, so you might also consider using Docker for your next paper replication material (for an example, see this repo).
An important extension of Docker is Docker Compose, which facilitates the common task of running multiple containers in concert. A composition is launched and managed by running commands in the same directory as a compose file. Compose files let you define which images to use, whether launched containers should be networked together, whether they should share storage directories, as well as letting you set environment variables and other settings. Docker Swarm is a further extension, which includes self-healing, scaling, and multi-host capabilities.
A good book about Docker is Docker in Action.
ViEWS 3 is a collection of (mostly) ASGI Apps. ASGI is a calling convention for running servers written in Python, which extends its predecessor WSGI with support for asyncronous handling of requests (which makes it fast). The ASGI implementation, or web server, used by ViEWS 3 for running python services is called uvicorn.
While deep knowledge of uvicorn is not necessary for ViEWS 3 maintenance, maintainers should be familiar with its configuration options, which are shown in the above link. For the purposes of remaining DRY, all ViEWS 3 uvicorn-driven apps have a common configuration scheme defined in the uvicorn_deployment repository, in which a parent Docker image is defined from which all uvicorn-based services should inherit.
The ubiquitous versioning program Git is of course also used here. There are many good Git tutorials, which you should definitely go through regardless of your work on this project: Git is extremely useful for academic work, including writing, replication packages, et cetera.
The definitive guide and reference for Git is Pro Git, which is available online for free.
One of the more obscure Git features, which is used for the views3 metarepository is git submodules. Git submodules allows you to nest repositories inside other repositories. Each service lives in a separate repository, so the repository containing the entire system must necessarily contain other repositories. Here, a metarepository is just defined as a repository containing other repositories.
Working with ViEWS 3, what you need to know about submodules is how they work
when cloning / pulling a repository. By default, cloning a metarepository does
not clone subrepositories. To do that, you need to explicitly tell git to
recurse when cloning, by adding the --recurse-submodules flag, like so:
git clone --recurse-submodules http://github.com/prio-data/views3
Note that you only need the subrepositories for this repo if you are planning to develop the services, the docker-compose file lets you spin up the services without having to clone the code, since services are stored as images on ACR.
If you've cloned a metarepo without this flag, you can still pull all of the subrepositories by running these commands in the metarepo:
git submodule update --init
This should clone each subrepository into their respective folders.
Continous integration / Continuous deployment is a term covering various practices for automating code deployment. Simply put, CI/CD involves making machines run the often repetetive and error-prone processes necessary to deploy and run code as applications.
For teams, CI/CD is extremely valuable, since it involves codifying processes that may often be obscure, understood only by "operators" responsible for deployments. With CI / CD, these processes are stated as code, and may be versioned along with the code.
ViEWS uses, and will extend its use of CI/CD to automate deployments, as far as possible. Github Actions is our CI/CD tool of choice, since it is first-class integrated with Github, where ViEWS 3's code is hosted. Github Actions can be triggered by repository pushes, making it possible to launch builds (such as docker image builds) whenever new code is pushed, ensuring that downstream build products remain up to date. This eliminates ambiguity about code -> build, ensuring a predictable relationship between built and stored products, and code.