VEuPathDB Dataset Installer Service (VDI)

Table of Contents

• Tagging, Deployment, and Branch Rules
  • Branch Rules
  • Tagging
  • Deployment
• Links
  • Documentation
  • VDI Project Index
• Repo Structure
  • Internal Libs
  • Daemons
  • Lanes
  • Rest Service
  • Bootstrapper
• Development
  • Running the Stack

Tagging, Deployment, and Branch Rules

Branch Rules

qa

The qa branch is to reflect what is currently on QA. Code may only be pushed to this branch via pull request which depends on both approval and a successful workflow build of the code.

production

The production branch should always reflect what is on production. This branch is to remain locked until we are ready to move code from QA to production, at which point the following steps will be performed:

1. The production branch will be unlocked via the repository settings.

2. The qa branch will be merged into production

3. The production branch will be re-locked via the repository settings.

❗	These steps are not the service deployment steps, they are simply an outline of steps strictly related to getting the production branch up to date with the qa branch. For deployment procedure steps see the Deployment section below.

main

The main branch is for active development. It is strongly advised that fixes and features be developed on branches forked from main and then merged back into main when complete to avoid conflict problems with other active development.

Tagging

Release tags v* are only to be made on the production branch.

Other tags should be avoided.

Deployment

The following is a full list of steps necessary to perform a deployment of the QA VDI service code onto production.

1. Unlock the production branch via the repository settings. This setting can be found in the "Branches" menu under the "production" branch rule. The specific setting to uncheck is "Lock Branch".

2. Merge the qa branch into production.

3. Re-lock the production branch via the setting described in step 1.

4. Wait for the production branch to build in Jenkins.

5. Create a new release tag on the merge commit from qa to production.

6. Wait for the new tag to build in Jenkins.

7. Create a PR on the Tagger repository, forked from and targeting the prod branch, updating the vdi-service tag to the new tag created in step 5.

8. Assign the PR to systems and notify them of the PR.

⚠️	Steps 4 and 6 are necessary to avoid known issues with our current Jenkins build setup for containerized services.

Links

Documentation

• VDI Service API Doc

• Service Design Doc

• Original Design Doc

• Environment Config Doc

• Local Development Doc

VDI Project Index

Services

• VDI Core Service

• VDI Plugin Handler Service

Docker Images

• Cache DB Docker Image

• Plugin Handler Base

Libraries

Service Libraries

• Commons Library

• JSON Utilities

Plugin Libraries

• lib-rnaseq

• lib-study

Plugins

Production

• bigwig

• biom

• isasimple

• rnaseq

• genelist

Examples

• Example Plugin

Repo Structure

The VDI service repository is divided into 4 categories: components, daemons, lanes, and the core bootstrapper.

Components are small shared libraries that contain code specific to a purpose that is not the core focus of the VDI service.

Daemons are background processes that perform maintenance tasks such as dataset reconciliation or pruning old, soft-deleted datasets from S3.

Lanes are event handlers that are tailored to individual event types, performing some process or processes on a dataset specified in the subscribed events.

The bootstrapper is the core of the service, and the root level Gradle project. This piece is responsible for starting up and shutting down the various project modules.

Internal Libs

Shared library components used by one or more VDI daemons, lanes, the rest service, and/or the bootstrapper.

• App DB Client

• Async Utilities

• Broken Install Cleanup

• Cache DB Client

• Commons

• Dataset Pruner

• Dataset Reinstaller

• JSON Utilities

• Kafka Client

• LDAP Utilities

• Module Core

• Plugin Server Client

• Plugin Mapping

• Rabbit Client

• S3 Utilities

• Shared Env Constants & Utilities

• Unit Test Utilities

Daemons

Background tasks that run unsupervised.

• Dataset Reinstaller

• Event Router

• Pruner

• Reconciler

Lanes

Dataset event handlers. Each lane is a separate process that subscribes to a Kafka channel and operates on datasets whose information is provided in the incoming events.

• Hard Delete Event Handler

• Import Event Handler

• Install Data Event Handler

• Reconciliation Event Handler

• Share Event Handler

• Soft Delete Event Handler

• Update Meta Event Handler

Rest Service

The rest service is the public API through which users and administrators communicate with and operate on the VDI system.

• Rest API Service

Bootstrapper

The bootstrapper is the core of the service, and the root level Gradle project. This piece is responsible for starting up and shutting down the various project modules.

• Bootstrapper

Development

Running the Stack

1. make build
   Builds the VDI service docker image.

2. make up
   Spins up the service.

3. make down
   Shuts down the service and removes all the vdi-specific containers, volumes, and networks.