Developer Recipes
PhenoMeNal Developer Recipes are a collection of scripts/tricks to make contributing and testing in the PhenoMeNal project a little bit easier. Feel free to add your own!
- Minikube with local Galaxy runtime recipe
- Quick rebuild and start helm script
- Testing your tool containers
- Developing tools with Planemo and phnmnl/galaxy
Added by David. Based on scripts from Pierrick.
Commands here are used in OSX. Might be similar in Linux, but not sure.
When developing tools for PhenoMeNal, testing them in container-galaxy-k8s-runtime on minikube is needed to make sure the tool wrappers works well with the tool containers. The commands here can help with setting up running a local Galaxy instance inside k8s.
A lot of this is already described in QuickStart Installation for Local PhenoMeNal Workflow, but I found these steps below much easier to digest.
- Installed minikube
- Installed helm
- Checked out of GitHub container-galaxy-k8s-runtime repository
Clean up minikube:
minikube delete
rm -r $HOME/.helm
minikube start
minikube ssh
sudo rm -rf /data/galaxy-data
exit
Open minikube dashboard in your browser with:
minikube dashboard
Build container-galaxy-k8s-runtime (make sure you run this from inside where you checked out the container's project, otherwise the docker build bit will not work:
helm init
helm repo add galaxy-helm-repo https://pcm32.github.io/galaxy-helm-charts
eval $(minikube docker-env)
docker build -t my-galaxy .
Install and run on minikube. This runs the image you built and tagged my-galaxy:latest but with the tools in development found in /Users/Alice/Development/container-galaxy-k8s-runtime. Make sure you update this path to where your container-galaxy-k8s-runtime was checked out.
For current Cerebellin release:
helm install --set galaxy_image="my-galaxy",galaxy_image_registry="",pv_minikube="yes",galaxy_pull_policy="IfNotPresent",galaxy_admin_email="a@b.com",galaxy_admin_password="pheno123",galaxy_api_key="qwertyuio" galaxy-helm-repo/galaxy
OR for Galaxy 18.01 migration:
helm install --set galaxy_image="my-galaxy",galaxy_image_tag="",galaxy_image_registry="",pv_minikube="yes",galaxy_pull_policy="IfNotPresent",log_file_to_track="galaxy.log" galaxy-helm-repo/galaxy
It takes a while for Galaxy to start. You can open up the Galaxy instance being launched in your browser with:
open http://$(minikube ip):30700
If you want to check the status of your Galaxy being launched, you can do:
kubectl get pods # to get the pod name. Should be something like galaxy-k8s-ahash
kubectl logs -f galaxy-k8s-ahash # replace galaxy-k8s-ahash with your galaxy pod
to view the live logs being written to stdout.
Please note! If your tools are not executed, and remain waiting in Galaxy you should check the Galaxy pod log file as described above. It may be the permissions are not set correctly, if so please run the following commands:
kubectl delete --ignore-not-found clusterrolebinding permissive-binding
kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts
If you want to restart your Galaxy runtime container, do somethings like:
helm list # to get your instance name
helm delete grumpy-goat # or something similar
kubectl delete secrets/galaxy-admin-secret
kubectl delete configmaps/galaxy-admin-user
kubectl delete configmaps/db-connection
kubectl delete secrets/galaxy-postgres-secret
Build your container image again and restart with helm install as before.
I made a script called quick_restart.sh that compliments the above recipe, so when testing the Galaxy runtime I just run the script to kill the running helm release, secrets that need resetting, rebuild the Galaxy docker image, and then run helm install again.
helm ls --short | xargs -L1 helm delete
kubectl delete secrets/galaxy-admin-secret
kubectl delete configmaps/galaxy-admin-user
docker build -t my-galaxy .
helm install --set galaxy_image="my-galaxy",galaxy_image_registry="",pv_minikube="yes",galaxy_pull_policy="IfNotPresent",galaxy_admin_email="a@b.com",galaxy_admin_password="pheno123",galaxy_api_key="qwertyuio" galaxy-helm-repo/galaxy
See https://gist.github.com/djcomlab/7ebdafbbfedf80870a9ba278c192b0bf
Added by David.
Might be obvious, but I always forget how to run my container tests when developing my tool containers.
- Docker
- Your tool container repo, e.g.
phnmnl/container-my-tool
From your tool container project directory, do a local docker build and tag it with something easy to remember.
docker build -t container-my-tool .
Add --no-cache if an upstream base image has been updated and you have previously built locally (and may have cached a tagged image already).
docker build --no-cache -t container-my-tool .
To run the tests that you have defined in your tool container, do a docker run but with the entrypoint pointing to your test script.
docker run --entrypoint "run_test.sh" container-my-tool
Added by David.
When developing tools for PhenoMeNal, testing them in container-galaxy-k8s-runtime on can be a bit heavy. Ideally we should be able to develop our Galaxy tools in vanilla Galaxy first using Planemo. However, at the moment if we want to develop tools against the ISA datatypes (isa-tab and isa-json) we have to rely on our own flavour of Galaxy with the datatype implementation.
- Installed planemo
- Checked out of GitHub phnmnl/galaxy repository
The most straightforward way to use Galaxy with the ISA datatype is to checkout the latest PhenoMeNal Galaxy release, at time of writing is Cerebellin.
cd /path/to/phnmnl/galaxy
git checkout release_17.09_plus_isa_k8s_resource_limts
You will need to add the following line to the pinned-requirements.txt that is hard-linked to requirements.txt in the root directory of the galaxy Git repository that you checked out:
isa-rwval==0.10.0
This is missing because in the container-galaxy-k8s-runtime project it installs isa-rwval via Docker rather than in the Galaxy requirements.txt.
Finally, go back to your tool development directory, and run Planemo pointing to the Galaxy source using the --galaxy_root parameter:
cd /path/to/my/tools/
planemo serve --galaxy_root /path/to/phnmnl/galaxy --no_cache_galaxy
This recipe may not be needed once the ISA datatype implementation has been successfully pulled into the main Galaxy project.
Add your recipes at the bottom of the page (please leave this notice here though)!
Don't forget to add it to the list of recipes at the top of the page.
 |
Funded by the EC Horizon 2020 programme, grant agreement number 654241 |  |
|---|
