Galaxy Kubernetes FAQ

Q: Galaxy spins up correctly on Kubernetes but I get this error on the Galaxy logs every time I try to send a job from Galaxy:

requests.packages.urllib3.connectionpool DEBUG 2018-06-12 11:01:49,163 "GET /apis/batch/v1/namespaces/default/jobs/galaxy-1 HTTP/1.1" 403 309
galaxy.jobs.runners ERROR 2018-06-12 11:01:49,164 (1) Unhandled exception calling queue_job
Traceback (most recent call last):
  File "/galaxy/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
    method(arg)
  File "/galaxy/lib/galaxy/jobs/runners/kubernetes.py", line 117, in queue_job
    if job.exists() and not self._galaxy_instance_id:
  File "/galaxy/.venv/local/lib/python2.7/site-packages/pykube/objects.py", line 87, in exists
    self.api.raise_for_status(r)
  File "/galaxy/.venv/local/lib/python2.7/site-packages/pykube/http.py", line 106, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
HTTPError: jobs.batch "galaxy-1" is forbidden: User "system:serviceaccount:default:default" cannot get jobs.batch in the namespace "default"

Usually these messages can be retrieved by running

supervisorctl tail -5000 galaxy:handler0 stderr

in the Galaxy pod on one of the job handlers (here: handler0, alternative: handler1)

A: This probably a misconfiguration of the RBAC settings. Make sure that you called the helm repo (galaxy or galaxy-stable) with rbac_needed=true variable.

Q: When trying to use RBAC support for galaxy-stable helm chart on minikube (setting rbac_needed: true I get the following error:

Error: UPGRADE FAILED: failed to create resource: roles.rbac.authorization.k8s.io "galaxy-role-pods-jobs" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods/log"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["pods/log"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods/log"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["jobs"], APIGroups:["batch"], Verbs:["get"]} PolicyRule{Resources:["jobs"], APIGroups:["batch"], Verbs:["list"]} PolicyRule{Resources:["jobs"], APIGroups:["batch"], Verbs:["watch"]} PolicyRule{Resources:["jobs"], APIGroups:["batch"], Verbs:["create"]} PolicyRule{Resources:["jobs"], APIGroups:["batch"], Verbs:["update"]} PolicyRule{Resources:["jobs"], APIGroups:["batch"], Verbs:["patch"]} PolicyRule{Resources:["jobs"], APIGroups:["batch"], Verbs:["delete"]} PolicyRule{Resources:["jobs"], APIGroups:["extensions"], Verbs:["get"]} PolicyRule{Resources:["jobs"], APIGroups:["extensions"], Verbs:["list"]} PolicyRule{Resources:["jobs"], APIGroups:["extensions"], Verbs:["watch"]} PolicyRule{Resources:["jobs"], APIGroups:["extensions"], Verbs:["create"]} PolicyRule{Resources:["jobs"], APIGroups:["extensions"], Verbs:["update"]} PolicyRule{Resources:["jobs"], APIGroups:["extensions"], Verbs:["patch"]} PolicyRule{Resources:["jobs"], APIGroups:["extensions"], Verbs:["delete"]}] user=&{system:serviceaccount:kube-system:default bb29c816-6e57-11e8-a2aa-080027376711 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[]

A: You might need to setup helm's tiller pod with RBAC support first (grant it cluster-admin role): https://github.com/jetstack/cert-manager/issues/256#issuecomment-358362665

Q: How do I check the free disk space left in my deployment:

Run df in the galaxy pod:

 $ kubectl exec -it <the galaxy pod name> -- /bin/df -h | grep gluster
 192.168.0.23:gluster  180G  169G   12G  94% /export