Merge pull request #24 from MS-Quality-Hub/v1.0.0rc1

Merging the release branch for v1.0.0 RC1, moving from MS-Quality-Hub/v1.0.0rc1 to MS-Quality-Hub/v1.0.0rc2

.devcontainer/Dockerfile

@@ -1,4 +1,5 @@
-FROM python:3.6.9-slim-buster
+FROM python:3.8-slim-buster
+#FROM python:3.6.9-slim-buster
 #FROM python:3.6.5-slim-jessie
 #FROM python:2.7-slim-jessie
 
@@ -24,7 +25,8 @@ RUN echo "deb http://http.debian.net/debian buster main" > /etc/apt/sources.list
 
 # # only packages related to dev, project dependencies will get resolved via pip and setup.py
 RUN python3 -m pip install --upgrade pip 
-RUN pip install matplotlib pytest mypy types-setuptools sphinx build wheel twine
+RUN pip install matplotlib pytest mypy types-setuptools sphinx build wheel twine click flask flask_restful flask_cors
+
 #RUN pip install -e .  # devcontainer.json: "postCreateCommand": "pip install -e .",
 
 ARG USERNAME=vscode

.github/workflows/pullrequest_builds.yml

@@ -9,7 +9,8 @@ jobs:
     strategy:
       max-parallel: 4
       matrix:
-        python-version: [3.6, 3.7, 3.8]
+        # python-version: [3.8, 3.9, '3.10']
+        python-version: [3.8]
 
     steps:
     - uses: actions/checkout@v1 

.github/workflows/release_builds.yml

@@ -9,7 +9,7 @@ jobs:
     strategy:
       max-parallel: 4
       matrix:
-        python-version: [3.6, 3.7, 3.8]
+        python-version: [3.8]
 
     steps:
     - uses: actions/checkout@v1 

.github/workflows/unit_tests.yml

@@ -8,6 +8,7 @@ jobs:
     strategy:
       max-parallel: 4
       matrix:
+        # python-version: [3.8, 3.9, '3.10']
         python-version: [3.8]
 
     steps:

.gitignore

@@ -1,4 +1,5 @@
 
+__pycache__/
 mzqc/__pycache__/
 tests/__pycache__/
 .mypy_cache/

BUILD.md

@@ -0,0 +1,73 @@
+# How to build and release
+(aka pre-filght tests for a release)
+
+## Manual build and release instruction
+
+First, install a local version of the release candidate installed via `pip git+` and get the sources for test and build, too:
+```bash
+    cd /tmp
+    python3 -m venv pipgit && source pipgit/bin/activate
+    pip install pip --upgrade
+    pip install pytest build
+    pip install -U git+https://github.com/MS-Quality-hub/pymzqc.git@v1.0.0rc1#egg=pymzqc
+    git clone --single-branch --branch=v1.0.0rc1 --depth=1 https://github.com/MS-Quality-hub/pymzqc.git
+```
+
+Re-activate your venv to let pytest reset to current venv and test installation: 
+```bash
+    deactivate && source pipgit/bin/activate
+    cd /tmp/pymzqc
+    pytest
+```
+
+Then, provide container-based builds an existing dist folder setup like so, and build:
+```bashcd 
+    cd /tmp/pymzqc
+    mkdir -p dist/mzqc
+    python3 -m build --sdist
+    python3 -m build --wheel
+```
+The build results will be at `/tmp/pymzqc/dist`. We'll need both as release artifacts.
+Now install the wheel in a new venv and test the wheel:
+```bash
+    cd /tmp/pymzqc
+    deactivate
+    python3 -m venv pipwhl && source pipwhl/bin/activate
+    pip install pip --upgrade
+    pip install pytest wheel dist/pymzqc-1.0.0rc1-py3-none-any.whl
+    deactivate && source pipwhl/bin/activate
+    cd /tmp/pymzqc
+    pytest
+```
+
+Also test wheel installation in legacy mode (w/o wheel module installed):
+```bash
+    cd /tmp/pymzqc
+    deactivate
+    python3 -m venv pipwhl && source pipwhl/bin/activate
+    pip install pip --upgrade
+    pip install pytest dist/pymzqc-1.0.0rc1-py3-none-any.whl
+    deactivate && source pipwhl/bin/activate
+    cd /tmp/pymzqc
+    pytest
+```
+
+If all tests were successful, upload to test.pypi.org with twine:
+```bash
+    twine upload --repository-url https://test.pypi.org/legacy/ dist/*
+```
+
+And finally test pypi installation:
+```    cd /tmp/pymzqc
+    deactivate
+    python3 -m venv pippypi && source pippypi/bin/activate
+    pip install pip --upgrade
+    pip install pytest
+    pip install -i https://test.pypi.org/simple/ pymzqc==1.0.0rc1
+    deactivate && source pippypi/bin/activate
+    cd /tmp/pymzqc
+    pytest
+```
+
+Check the release label isn't already in use in pypi.org/pymzqc (otherwise correct,repeat build & test).
+Now upload to pypi.org, fingers crossed.

CONTRIBUTING.md

@@ -0,0 +1,75 @@
+# Contribute to mzQC
+
+Thank you for investing your time in contributing to our project! :sparkles:. 
+
+Read the [Code of Conduct](https://github.com/github/docs/blob/main/CODE_OF_CONDUCT.md) to keep our community approachable and respectable. (We adopted GitHub's open source code of conduct.)
+
+## Contribute Your Metrics
+
+We welcome every metric contribution, be it new or established. The easiest way is to open a new issue on the PSI-MS CV repository with our `New QC Term` template. You can read more detail about the process on the [Metrics page](metrics).
+
+## New contributor guide
+
+To get an overview of the project, read the [README](README.md). Here are some general resources to help you get started with open source contributions on GitHub:
+
+- [Finding ways to contribute to open source on GitHub](https://docs.github.com/en/get-started/exploring-projects-on-github/finding-ways-to-contribute-to-open-source-on-github)
+- [Set up Git](https://docs.github.com/en/get-started/quickstart/set-up-git)
+- [GitHub flow](https://docs.github.com/en/get-started/quickstart/github-flow)
+- [Collaborating with pull requests](https://docs.github.com/en/github/collaborating-with-pull-requests)
+
+
+## Getting started
+
+A good starting point is our website to get informed on the format and it's mechanistics, how to get in touch with us, ongoing issues, and related projects.
+
+### Issues
+
+We use GitHub's issue tracking system to stay on-top of new developments and arising issues.
+
+#### Create a new issue
+
+If you spot a problem, [search if an issue already exists](https://docs.github.com/en/github/searching-for-information-on-github/searching-on-github/searching-issues-and-pull-requests#search-by-the-title-body-or-comments). If a related issue doesn't exist, you can open a new issue using a relevant [issue form](https://github.com/github/docs/issues/new/choose). 
+
+#### Solve an issue
+
+Scan through our [existing issues](
+hupo-psi.github.io/mzqc/ ) to find one that interests you. You can narrow down the search using `labels` as filters. See [Labels](https://github.com/github/docs/contributing/how-to-use-labels.md) for more information. As a general rule, we don’t assign issues to anyone. If you find an issue to work on, you are welcome to open a PR with a fix.
+
+#### Request a new metric
+
+If you miss a specific metric, you can request it's addition via the issue system. It might be good if you'd familiarise yourself with [mzQC](https://hupo-psi.github.io/mzqc/) and [CV metric representation](https://hupo-psi.github.io/mzqc/metrics/) in particular. We have a dedicated issue template for the request of new QC terms in the PSI-MS-CV repository. 
+
+### Make Changes
+
+1. Fork the repository.
+2. Create a working branch and start with your changes!
+3. Commit your update(s).
+
+#### Pull Request
+
+When you're finished with the changes, create a pull request, also known as a PR.
+- Fill the "Ready for review" template so that we can review your PR. This template helps reviewers understand your changes as well as the purpose of your pull request. 
+- Don't forget to [link PR to issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue) if you are solving one.
+- Enable the checkbox to [allow maintainer edits](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork) so the branch can be updated for a merge.
+Once you submit your PR, a team member will review your proposal. We may ask questions or request for additional information.
+- We may ask for changes to be made before a PR can be merged, either using [suggested changes](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/incorporating-feedback-in-your-pull-request) or pull request comments. You can apply suggested changes directly through the UI. You can make any other changes in your fork, then commit them to your branch.
+- As you update your PR and apply changes, mark each conversation as [resolved](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/commenting-on-a-pull-request#resolving-conversations).
+- If you run into any merge issues, checkout this [git tutorial](https://github.com/skills/resolve-merge-conflicts) to help you resolve merge conflicts and other issues.
+
+#### Labels
+
+Labels can help you (*and us*) find and organise issues more readily. Here are some labels explained:
+- The help wanted label is for problems or updates that anyone in the community can start working on.
+- The good first issue label is for problems or updates we think are ideal for beginners.
+- The schema label is for problems or updates to the mzQC format itself. These are considered with the outmost care and detail.
+- The documentation labels (specification document and website) label issues found in the specification document or with documentation content (such as the website)
+- The CV label is for issues and discussion about the metrics and our CV integration.
+- obsolete: request for new CV entry (We merged with PSI-MS-CV, see labels there.) 
+
+#### Your PR is merged!
+
+Congratulations :tada::tada: The mzQC team thanks you :sparkles:. 
+
+Once your PR is merged, your contributions will be publicly visible on the [repo](https://github.com/HUPO-PSI/mzQC). 
+
+

QUICKSTART.md

@@ -0,0 +1,36 @@
+# Usage examples
+I am a fan of quick hands-on overviews to get to know software, that I might or might not use in the future.
+Several hands-on examples can be found [here](https://github.com/MS-Quality-hub/pymzqc/tree/main/jupyter)
+Simply open them from [colab](https://colab.research.google.com/)
+A video version for ASMS'22 is `on [youtube](https://www.youtube.com/watch?v=vZXJuPl2yGw)
+
+**The copy&paste essentials:**
+
+## Load
+```python
+    from mzqc import MZQCFile as qc
+    with open("nameOfYourFile.mzQC", "r") as file:
+        my_run_qualities = qc.JsonSerialisable.FromJson(file)
+```
+
+## Access elements
+see [schema](https://github.com/HUPO-PSI/mzQC/tree/main/schema>) for a general overview of available elements.
+```python
+    # An in-memory mzQC file will still have the same hierarchical structure as the schema
+    print(my_run_qualities.description)
+
+    # JSON arrays can be used like python lists
+    for m in my_run_qualities.qualityMetrics:
+        print(m.name)
+
+    # You can traverse the hierarchy with standard python member access notation ('.') 
+    # and get to the bottom of things (like a metric value).
+    ms2_number = my_run_qualities.qualityMetrics[2].value
+```
+
+## Store
+```python
+    inmem_file = qc.JsonSerialisable.ToJson(mzqc, readability=1)
+    with open("nameOfYourFile.mzQC", "w") as file:
+        file.write(inmem_file)
+```

accessories/heroku/Dockerfile

@@ -0,0 +1,13 @@
+FROM python:3.8-slim-buster
+
+ENV DEBIAN_FRONTEND noninteractive 
+
+ENV LANG C.UTF-8
+ENV LC_ALL C.UTF-8
+
+COPY mzqc_online_validator.py mzqc-validator/
+COPY requirements.txt mzqc-validator/
+COPY Procfile mzqc-validator/
+
+RUN python -m pip install --upgrade pip
+RUN pip install -r mzqc-validator/requirements.txt 

accessories/heroku/README.md

@@ -0,0 +1,33 @@
+### Simple API
+
+The simple API has three endpoints: 
+1. to indicate '/status/' (GET)
+2. providing '/documentation/' (GET) 
+3. to post `mzQC` files to '/validator/' (POST)
+
+The documentation endpoint provides a `dict` with details to each part of the validation (key) as text (value).
+The validator endpoint takes a mzQC file (JSON) and responds with an object as described in the documentation endpoint.
+
+### Local Validation & Testing
+For local validation and validator development tests, you can start a local validation API like so:
+```
+python3 -m venv /tmp/vval
+source /tmp/vval/bin/activate
+pip install -r accessories/heroku/requirements.txt
+python accessories/heroku/mzqc_online_validator.py
+```
+e.g. with `accessories/heroku/local_validator.html`
+You can find both files necessary within the repository under [accessories/heroku](https://github.com/MS-Quality-hub/pymzqc/tree/main/accessories/heroku).
+
+### Deploy
+Or you can deploy your own heroku dyno like so:
+```
+cd /tmp/
+curl https://cli-assets.heroku.com/install-ubuntu.sh | sh
+heroku login
+heroku git:clone -a mzqc-validator 
+cd mzqc-validator
+rsync -aP --delete /home/walzer/psi/pymzqc/acessories/heroku  /tmp/mzqc-validator
+git push heroku master
+```
+

accessories/heroku/dev-test validation.py

@@ -0,0 +1,95 @@
+# dev-test validation
+from flask import Flask
+from flask import Flask, jsonify, flash, redirect, url_for, request
+from werkzeug.utils import secure_filename
+from flask_restful import Resource, Api
+from flask_cors import CORS
+
+import json
+from mzqc.MZQCFile import MzQcFile as mzqc_file
+from mzqc.MZQCFile import JsonSerialisable as mzqc_io
+from mzqc.SemanticCheck import SemanticCheck
+from mzqc.SyntaxCheck import SyntaxCheck
+
+# temporarily ln -s /home/walzer/psi/mzQC-development/doc/examples tests/examples
+testfiles = ["tests/examples/individual-runs.mzQC",
+"tests/examples/metabo-batches.mzQC",
+"tests/examples/Mtb-120-outlier-metrics.mzqc",
+"tests/examples/QC2-sample-example.mzQC",
+"tests/examples/set-of-runs.mzQC",
+"tests/examples/USI-nativeID-example.mzQC"]
+
+# #https://flask.palletsprojects.com/en/2.0.x/patterns/fileuploads/
+# def upload_file():
+#     if request.method == 'POST':
+#         # check if the post request has the file part
+#         if 'file' not in request.files:
+#             flash('No file part')
+#             return redirect(request.url)
+#         file = request.files['file']
+#         # If the user does not select a file, the browser submits an
+#         # empty file without a filename.
+#         if file.filename == '':
+#             flash('No selected file')
+#             return redirect(request.url)
+#         if file and allowed_file(file.filename):
+#             filename = secure_filename(file.filename)
+#             file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
+#             return redirect(url_for('download_file', name=filename))
+#     return "blablabla"
+
+
+import json
+from mzqc.MZQCFile import MzQcFile as mzqc_file
+from mzqc.MZQCFile import JsonSerialisable as mzqc_io
+from mzqc.SemanticCheck import SemanticCheck
+from mzqc.SyntaxCheck import SyntaxCheck
+
+# infi = "tests/examples/individual-runs.mzQC"  # success test
+# infi = "tests/examples/individual-runs-no-outer.json"  # No mzQC content found! no mzQC object no detectin
+# infi = "tests/examples/individual-runs_extraJSONcontent.mzQC"  # test good detectin schema invalid, also QC:000 terms unknown
+# infi = "tests/examples/individual-runs_tableExtraColumn.mzQC"  # test good detectin
+# infi = "tests/examples/individual-runs_wrongTermName.mzQC"  # test good detectin
+# infi = "tests/examples/individual-runs_tableIncomplete.mzQC"  # test good detectin
+# infi = "tests/examples/individual-runs_brokenAnalysisSoftware.mzQC"  # test good detectin schema invalid
+# infi = "tests/examples/individual-runs_unequalTableCols.mzQC"  # test good detection
+# infi = "tests/examples/individual-runs_duplicateMetric.mzQC"  # test good detection
+infi = "tests/examples/individual-runs_non-metric-term.mzQC"  # test good detection
+
+with open(infi, 'r') as f:
+    inpu = f.read()
+# input from web validator comes in as a js dumps() equivalent
+
+try:
+    mzqcobject = mzqc_io.FromJson(inpu)
+    print("Found mzQC content, successfully loaded!")
+except Exception as e:
+    print("No mzQC content found!")
+    try:
+        json.loads(inpu)
+        dummy = "mzQC-incompatible JSON"
+        # dummy = json.loads(inpu)
+        # print("Found json content, loading failed!")
+        # print("did you forget the outer 'mzQC' JSON element?")
+    except:
+        # print("Unknown content, stop!")
+        ret = {'schema': 'abort due to unknown content', 'semantics': {'validation': 'abort due to unknown content'}}
+        #return jsonify(ret)
+
+ret = SyntaxCheck().validate(inpu)
+if type(mzqcobject) == mzqc_file:
+    removed_items = list(filter(lambda x: not (x.uri.startswith('http') or x.uri.startswith('file://')), mzqcobject.controlledVocabularies))
+    mzqcobject.controlledVocabularies = list(filter(lambda x: (x.uri.startswith('http') or x.uri.startswith('file://')), mzqcobject.controlledVocabularies))
+    sem_val = {'semantic validation': SemanticCheck().validate(mzqcobject, load_local=True)}
+    sem_val.update({'invalid ontology URIs': ' , '.join([str(x) for x in removed_items])})
+else:
+    sem_val = {'semantic validation': {'validation': 'abort due to mzQC-incompatible JSON'}}
+ret.update(sem_val)
+#return jsonify(ret)
+print(json.dumps(ret, indent=2, sort_keys=True))
+
+
+
+
+# SyntaxCheck().validate(json.dumps(purejson))['schema'][0].partition('\n')[0]  # old version of SyntaxChecker
+