Flask microservice implementing the Global Alliance for Genomics and Health (GA4GH) Workflow Execution Service (WES) API specification.
WES-ELIXIR is an implementation of the GA4GH WES OpenAPI specification based on Flask and Connexion. It allows clients/users to send workflows for execution, list current and previous workflow runs, and get the status and/or detailed information on individual workflow runs. It interprets workflows and breaks them down into individual tasks, for each task emitting a request that is compatible with the GA4GH Task Execution Service (TES) OpenAPI specification. Thus, for end-to-end execution of workflows, a local or remote instance of a TES service such as TESK or funnel is required.
The service is backed by a MongoDB database and optionally uses JWT token-based authorization, e.g. through ELIXIR AAI. While currently only workflows written in the Common Workflow Language are supported (leveraged by CWL-TES, we are planning to abstract workflow interpretation away from API business logic on the one hand and task execution on the other, thus hoping to provide an abstract middleware layer that can be interfaced by any workflow language interpreter in a pluggable manner.
Note that the project is currently still under active development. Nevertheless, a largely FUNCTIONAL PROTOTYPE is available as of October 2018, hosted at the CSC in Helsinki.
WES-ELIXIR is part of ELIXIR, a multinational effort at establishing and implementing FAIR data sharing and promoting reproducible data analyses and responsible data handling in the Life Sciences. Infrastructure and IT support are provided by ELIXIR Finland at the CSC, the TESK service is being developed and maintained by ELIXIR UK at the EBI in Hinxton, and WES-ELIXIR itself is being developed by ELIXIR Switzerland at the Biozentrum in Basel and the Swiss Institute of Bioinformatics.
Ensure you have the following software installed:
- Docker (18.06.1-ce, build e68fc7a)
- docker-compose (1.22.0, build f46880fe)
- Git (1.8.3.1)
Note: These are the versions used for development/testing. Other versions may or may not work.
Create data directory and required subdiretories
mkdir -p data/db data/output data/tmpClone repository
git clone https://github.com/elixir-europe/WES-ELIXIR.git appTraverse to app directory
cd appPlace a .netrc file for access to a FTP server in app directory.
Don't forget to replace <USERNAME> and <PASSWORD> with real values.
cat << EOF > .netrc
machine ftp-private.ebi.ac.uk
login <USERNAME>
password <PASSWORD>
EOFOptional: edit default and override app config
vi wes_elixir/config/app_config.yaml
vi wes_elixir/config/override/app_config.dev.yaml # for development service
vi wes_elixir/config/override/app_config.prod.yaml # for production serverBuild container image
docker-compose buildRun docker-compose services in detached/daemonized mode
docker-compose -f docker-compose.yaml -f docker-compose.dev.yaml up -d # for development service
docker-compose -f docker-compose.yaml -f docker-compose.prod.yaml up -d # for production serviceVisit Swagger UI
firefox http://localhost:7777/ga4gh/wes/v1/uiEnsure you have the following software installed:
- curl (7.47.0)
- Git (2.7.4)
- MongoDB (4.0.1)
- Python3 (3.5.2)
- RabbitMQ (3.5.7)
- virtualenv (16.0.0)
Note: These are the versions used for development/testing. Other versions may or may not work.
Ensure RabbitMQ is running (actual command is OS-dependent)
sudo service rabbitmq-server statusStart MongoDB daemon (actual command is OS-dependent)
sudo service mongod startPlace a .netrc file for access to a FTP server in your $HOME directory.
Don't forget to replace <USERNAME> and <PASSWORD> with real values.
cat << EOF > "${HOME}/.netrc"
machine ftp-private.ebi.ac.uk
login <USERNAME>
password <PASSWORD>
EOFClone repository
git clone https://github.com/elixir-europe/WES-ELIXIR.git appTraverse to project directory
cd app
project_dir="$PWD"Create and activate virtual environment
virtualenv -p `which python3` venv
source venv/bin/activateInstall required packages
pip install -r requirements.txtInstall editable packages
cd "${project_dir}/venv/src/cwl-tes"
python setup.py develop
cd "${project_dir}/venv/src/cwltool"
python setup.py develop
cd "${project_dir}/venv/src/py-tes"
python setup.py develop
cd "$project_dir"Install app
python setup.py developOptionally, override default config by setting environment variable and pointing it to a YAML config file. Ensure the file is accessible.
export WES_CONFIG=<path/to/override/config/file.yaml>Start service
python wes_elixir/app.pyIn another terminal, load virtual environment & start Celery worker for executing background tasks
# Traverse to project directory ("app") first
source venv/bin/activate
cd wes_elixir
celery worker -A celery_worker -E --loglevel=infoVisit Swagger UI
firefox http://localhost:8888/ga4gh/wes/v1/uiNote: If you have edited WES_CONFIG, ensure that host and port match the values specified in the config file.
Coming soon...
Join us at the 2018 BioHackathon in Paris, organized by ELIXIR Europe (November 12-16)! Check out our project description.
This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, or fixes and other code changes. Please read these guidelines if you want to contribute. And please mind the code of conduct for all interactions with the community.
Development of the app is currently still in alpha stage, and current "versions" are for internal use only. We are aiming to have a fully spec-compliant ("feature complete") version of the app available by the end of 2018. The plan is to then adopt a semantic versioning scheme in which we would shadow WES spec versioning for major and minor versions, and release patched versions intermittently.
This project is covered by the Apache License 2.0 also shipped with this repository.
The project is a collaborative effort under the umbrella of ELIXIR Europe.
Please contact the project leader for inquiries, proposals, questions etc. that are not covered by the Q&A and Contributing sections.
- https://www.elixir-europe.org/
- https://www.ga4gh.org/
- https://github.com/ga4gh/workflow-execution-service-schemas
See also krini-cwl for an older, more rudimentary, yet functional TES-independent WES implementation that is part of the Krini project and leverages Toil.