conformance

Calrissian - CWL Conformance Tests

This is under development, and will eventually be automated to run in a CI environment. However, the resources here can be used to run CWL conformance tests on a Kubernetes cluster.

The manifests are ready for a minikube local cluster.

Conformance tests for CWL v1.0 for the latest release

Classes

Required features

Optional features

Conformance tests for CWL v1.1 for the latest release

Classes

Required features

Optional features

Conformance tests for CWL v1.2.0 for the latest release

Classes

Required features

Optional features

Conformance tests for CWL v1.2.1 (proposed) for the latest release

Classes

Required features

Optional features

Run the conformance tests on Minikube

Cluster Preparation

To run the conformance tests, setup a namespace and create the resources needed with:

NAMESPACE_NAME=calrissian-conformance
kubectl create namespace "$NAMESPACE_NAME"
kubectl --namespace="$NAMESPACE_NAME" create role pod-manager-role \
  --verb=create,patch,delete,list,watch --resource=pods
kubectl --namespace="$NAMESPACE_NAME" create role log-reader-role \
  --verb=get,list --resource=pods/log
kubectl --namespace="$NAMESPACE_NAME" create rolebinding pod-manager-default-binding \
  --role=pod-manager-role --serviceaccount=${NAMESPACE_NAME}:default
kubectl --namespace="$NAMESPACE_NAME" create rolebinding log-reader-default-binding \
  --role=log-reader-role --serviceaccount=${NAMESPACE_NAME}:default

Staging Conformance Tests Data

To prepare a volume containing the conformance test data, create StageConformanceTestsData.yaml:

kubectl --namespace="$NAMESPACE_NAME" create -f StageConformanceTestsData.yaml

Building the container

This step can probably be simplified, but as conformance tests are in development it's too early to optimize

Build a container that installs calrissian and cwltest with:

minikube image build .. -t conformance:latest -f conformance/Dockerfile.conformance 

This will build calrissian:conformance from the current source tree and this image is available in the minikube cluster node.

Running Conformance Tests

ConformanceTestsJob-1.2.yaml uses cwltest from cwltool to run conformance tests with --tool calrissian and Calrissian's required arguments after --, e.g.:

...
  command: ["cwltest"]
  args:
  - "--test"
  - "/conformance/common-workflow-language-1.0.2/v1.0/conformance_test_v1.0.yaml"
  - "--tool"
  - "calrissian"
  - "--badgedir"
  - "/output/badges-1.0.2"
  - "--verbose"
  - "--"
  - "--max-ram"
  - "8G"
  - "--max-cores"
  - "4"
  - "--default-container"
  - "debian:stretch-slim"
...

CWL 1.0

kubectl --namespace="$NAMESPACE_NAME" create -f ConformanceTestsJob-1.0.yaml
kubectl --namespace="$NAMESPACE_NAME" wait --for=condition=Ready\
   --selector=job-name=conformance-tests-1-0 pods
kubectl --namespace="$NAMESPACE_NAME" logs -f jobs/conformance-tests-1-0 | egrep -v 'The `label`' > ../docs/conformance/1.0/results-v1.0.txt

CWL 1.1

kubectl --namespace="$NAMESPACE_NAME" create -f ConformanceTestsJob-1.0.yaml
kubectl --namespace="$NAMESPACE_NAME" wait --for=condition=Ready\
   --selector=job-name=conformance-tests-1-0 pods
kubectl --namespace="$NAMESPACE_NAME" logs -f jobs/conformance-tests-1-0

CWL 1.2.1

kubectl --namespace="$NAMESPACE_NAME" create -f ConformanceTestsJob-1.2.yaml
kubectl --namespace="$NAMESPACE_NAME" wait --for=condition=Ready\
   --selector=job-name=conformance-tests-1-2 pods
kubectl --namespace="$NAMESPACE_NAME" logs -f jobs/conformance-tests-1-2

Inspect the volumes content

Create a pod to inspect the volume with:

kubectl --namespace="$NAMESPACE_NAME" apply -f inspect-volumes-pod.yaml 

Open a shell in the pod with:

kubectl --namespace="$NAMESPACE_NAME" exec --stdin --tty inspect-volumes -- /bin/bash

Inspect the content of /output with ls -l /output

Copy the badges

Create a pod to copy the badges from the volume with:

kubectl --namespace="$NAMESPACE_NAME" apply -f inspect-volumes-pod.yaml 

kubectl cp $NAMESPACE_NAME/inspect-volumes:/output/badges-1.0/ badges/1.0

kubectl cp $NAMESPACE_NAME/inspect-volumes:/output/badges-1.1.0/ badges/1.1.0

kubectl cp $NAMESPACE_NAME/inspect-volumes:/output/badges-1.2.0/ badges/1.2.0

kubectl cp $NAMESPACE_NAME/inspect-volumes:/output/badges-1.2.1_proposed/ badges/badges-1.2.1_proposed

Notes:

• Calrissian requires specifying an envelope of RAM and CPU resources to use in the cluster, so these are provided as --max-ram and --max-cores
• Since kubernetes is entirely container-based, CWL Tools that do not specify a Docker image will not run unless Calrissian is run with --default-container
• This job uses an initContainer to guarantee the output volume is writable by the calrissian container. The calrissian image runs as a non-root user.
• This Job sets the TMPDIR environment variable to /outdir, which is backed by a persistent volume. This is because cwltest executes calrissian with --outdir set to a temp directory provided by Python's tempfile.mkdtemp(). By setting TMPDIR to a persistent volume backed-path, we meet Calrissian's requirement that output data is stored to a persistent volume.
• This Job sets the CALRISSIAN_POD_NAME variable using the Kubernetes Downward API. Calrissian needs to know its pod name at runtime so it can query the API for the names and mount points of persistent volume claims.