"/bin/bash: .command.run: Permission denied" with SELinux enabled

[junaruga]

Hi I faced the following "/bin/bash: .command.run: Permission denied" when running nf-core/nanoseq with with profile test,docker.

Is there any idea to find the reason? And is there a good way to debug?

Thank you.

My environment. nextflow is the latest version I installed last week.

The docker is installed by sudo dnf install moby-engine on Fedora 32.

$ uname -m
x86_64

$ cat /etc/fedora-release
Fedora release 32 (Thirty Two)

$ nextflow -v
nextflow version 20.07.1.5412

$ docker --version
Docker version 19.03.11, build 42e35e6

$ rpm -qf $(which docker)
moby-engine-19.03.11-1.ce.git42e35e6.fc32.x86_6

$ docker system prune -a -f

$ docker images 
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

$ pwd
/home/jaruga/git/nf-core/nanoseq

On master: b88e1c9a77083e90dd0bc5e900e9bcff84814559.

$ nextflow run nf-core/nanoseq -profile test,docker 2>&1 | tee nextflow-run-nanoseq-test-docker.log

Here is the log file.

Caused by:
  Process `output_documentation` terminated with an error exit status (126)

Command executed:

  markdown_to_html.py output.md -o results_description.html

Command exit status:
  126

Command output:
  (empty)

Command error:
  Unable to find image 'nfcore/nanoseq:1.0.0' locally
...
  Status: Image is up to date for nfcore/nanoseq:1.0.0
  /bin/bash: .command.run: Permission denied

Work dir:
  /home/jaruga/git/nf-core/nanoseq/work/7f/48ee820291acfcbfa6fc35f1c27139

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

The .command.out is empty.

$ cat  /home/jaruga/git/nf-core/nanoseq/work/7f/48ee820291acfcbfa6fc35f1c27139/.command.out

$ cat  /home/jaruga/git/nf-core/nanoseq/work/7f/48ee820291acfcbfa6fc35f1c27139/.command.sh 
#!/bin/bash -euo pipefail
markdown_to_html.py output.md -o results_description.html

[junaruga]

Seeing the /home/jaruga/git/nf-core/nanoseq/work/7f/48ee820291acfcbfa6fc35f1c27139/.command.run, I found the environment variable used for debug. Here is the log file.

It looks the error happens on the different part.

$ NXF_DEBUG=1 nextflow run nf-core/nanoseq -profile test,docker 2>&1 | tee nextflow-run-nanoseq-test-docker-debug.log
...
Command executed:

  git clone https://github.com/nf-core/test-datasets.git --branch nanoseq --single-branch

Command exit status:
  126
...
  + docker run -i --memory 6144m -e NXF_DEBUG=1 -v /home/jaruga:/home/jaruga -v /home/jaruga/git/nf-core/nanoseq/work/5e/b6a6a4be85fe31167de575fc0aa2d8:/home/jaruga/git/nf-core/nanoseq/work/5e/b6a6a4be85fe31167de575fc0aa2d8 -w /home/jaruga/git/nf-core/nanoseq/work/5e/b6a6a4be85fe31167de575fc0aa2d8 --entrypoint /bin/bash -u 1000:1000 --name nxf-VKAUvCi30ASP1Ed89jy58fJm nfcore/nanoseq:1.0.0 -c 'eval export PYTHONNOUSERSITE="1"
  export PATH="/home/jaruga/.nextflow/assets/nf-core/nanoseq/bin:$PATH"; /bin/bash .command.run nxf_trace'
  /bin/bash: .command.run: Permission denied
...

The image is downloaded on local.

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nfcore/nanoseq      1.0.0               62f539637a3f        5 months ago        2.7GB

As my experience to test to run git clone https://github.com/nf-core/test-datasets.git --branch nanoseq --single-branch in a container with docker, the following commands are ok.

$ cat Dockerfile.utuntu
FROM ubuntu:groovy

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get -y update
RUN apt-get -y install git
RUN git clone https://github.com/nf-core/test-datasets.git --branch nanoseq --single-branch

$ docker build --rm -t test/ubuntu -f Dockerfile.utuntu .

$ docker run --rm -t test/ubuntu bash -cx "cd test-datasets && git remote -v"
+ cd test-datasets
+ git remote -v
origin  https://github.com/nf-core/test-datasets.git (fetch)
origin  https://github.com/nf-core/test-datasets.git (push)

[drpatelh]

Hi @junaruga ! Apologies for the late response. It appears I wasn't watching this repo for some reason. Did you mange to fix the issue above?

[drpatelh]

Closing this but feel free to re-open if the issue persists in version 1.1.0.

[junaruga]

@drpatelh Sorry for the late reply. Could you reproduce the error I reported, on your environment? Then did you confirm the issue was fixed o the version 1.1.0?

[junaruga]

Hopefully I will check the issue this Friday.

[drpatelh]

Thanks @junaruga. Unfortunately, I don't use Docker but could you maybe try deleting the image and re-running? Sorry, I closed the issue hoping it would be magically fixed. If you do still observe the same issue then please re-open and we can try and debug further. I have a strong suspicion its down to the Docker set-up.

@apeltzer @ewels any ideas why you would get the error message below:

Caused by:
  Process `output_documentation` terminated with an error exit status (126)

Command executed:

  markdown_to_html.py output.md -o results_description.html

Command exit status:
  126

Command output:
  (empty)

Command error:
  Unable to find image 'nfcore/nanoseq:1.0.0' locally
...
  Status: Image is up to date for nfcore/nanoseq:1.0.0
  /bin/bash: .command.run: Permission denied

Work dir:
  /home/jaruga/git/nf-core/nanoseq/work/7f/48ee820291acfcbfa6fc35f1c27139

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

[junaruga]

If you do still observe the same issue then please re-open and we can try and debug further. I have a strong suspicion its down to the Docker set-up.

I still observe the issue today. You can see the log for detail. I do not have the permission to reopen the ticket. Could you reopen it? It might depends on the Docker setup on Fedora 32 or the Docker version. I did setup it seeing this comment by person from Docker. docker/for-linux#955 (comment) .

[drpatelh]

Re-opened the issue @junaruga and invited you to become a member of the nf-core organisation. Hopefully, this means you will be able to re-open issues in the future as well as other perks for being a member.

I am afraid I still have no idea why you are experiencing this issue and as you suggested may be related to the OS. The pipeline definitely works with Docker because all of our CI tests are run via Github Actions use Docker.

[drpatelh]

Could be worth asking on the nf-core Slack on the #help channel.

[junaruga]

Thanks for that. Yes. I saw the CI tests passing. I might ask it on Slack.
Anyway, I think keep opening this ticket is useful for people trying to run on Fedora & Docker.

[junaruga]

Here is the result log by the following command with podman. Same error happened.

$ nextflow run nf-core/nanoseq -profile test,podman 2>&1 | tee nextflow-run-nanoseq-test-podman.log

@drpatelh Do you know if the nextflow or nf-core/nanoseq is using the container's volume mount internally?

[junaruga]

In my environment, the SELinux is enabled with the mode: enforcing. I remember Ubuntu disables SELinux as a default setting.
I remember I needed :Z for the used volume mounted file path in the SELinux environment. You can see this manual for detail.

$ getenforce
Enforcing

$ sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      32

[junaruga]

I disabled the SELinux on my environment to check if it works on the SELinux disabled environment.

Change SELINUX=enforcing to SELINUX=disabled.

$ sudo vi /etc/selinux/config

$ sudo reboot

$ getenforce
Disabled

$ sestatus 
SELinux status:                 disabled

The error message /bin/bash: .command.run: Permission denied was disappeared. So, first we need to find which part of nextflow or nf-core/nanoseq is needed to be fixed to run the containers in SELinux enabled environment.

And now I see a new error ERROR: Please check samplesheet header -> group,replicate,barcode,input_file,genome,transcriptome != sample,fastq,barcode,genome,transcriptome. Here are the logs: docker, podman.

Do you know what's the reason?
Thanks!

[drpatelh]

The error message /bin/bash: .command.run: Permission denied was disappeared. So, first we need to find which part of nextflow or nf-core/nanoseq is needed to be fixed to run the containers in SELinux enabled environment.

Nice!! I think this will have to be queried on the Nextflow end. Someone may have come up against something similar or it could just be the wonderfulness of SELinux 🤦

Ok. We are finally getting somewhere though. Can you run nextflow pull nf-core/nanoseq and then try again?

This will pull and run the latest version of the pipeline locally. We have updated the format of the input samplesheets to the pipeline which is why I suspect it is failing.

[junaruga]

Nice!! I think this will have to be queried on the Nextflow end. Someone may have come up against something similar or it could just be the wonderfulness of SELinux

OK. I will try to ask on the Nextflow end.

Ok. We are finally getting somewhere though. Can you run nextflow pull nf-core/nanoseq and then try again?

Sure. let me check it.

[junaruga]

Sure. let me check it.

Finally I confirmed it worked on SELInux disabled environment! Thanks! Here are the logs: docker, podman.

Please keep opening this ticket for now until we will see the issue fixed on the SELinux enabled enviornment.

[junaruga]

I got the help on the Slack help channel.

you can specify that using mountFlags options
https://www.nextflow.io/docs/latest/config.html#config-docker

[drpatelh]

Sweet! So you have it working now? Be great if you can post the additional options you needed to use here too.

[junaruga]

I have not tried the options on SELinux enabled environment yet. I will share it after trying it.

[junaruga]

I tested it on the SELinux enabled enviornment. But here are the the definitions of the z and Z.

https://docs.docker.com/engine/reference/commandline/run/

Labeling systems like SELinux require that proper labels are placed on volume content mounted into a container. Without a label, the security system might prevent the processes running inside the container from using the content. By default, Docker does not change the labels set by the OS.

To change the label in the container context, you can add either of two suffixes :z or :Z to the volume mount. These suffixes tell Docker to relabel file objects on the shared volumes. The z option tells Docker that two containers share the volume content. As a result, Docker labels the content with a shared content label. Shared volume labels allow all containers to read/write content. The Z option tells Docker to label the content with a private unshared label. Only the current container can use a private volume.

Here is the result.

• docker.mountFlags = 'Z' showed an error /bin/bash: .command.run: Permission denied.
• podman.mountFlags = 'Z' showed an error Error: relabel failed "/home/jaruga": SELinux relabeling of /home/jaruga is not allowed. (the log)
• docker.mountFlags = 'z' worked by following config file.
• podman.mountFlags = 'z' showed an error by following config file Error: relabel failed "/home/jaruga": SELinux relabeling of /home/jaruga is not allowed. (the log). Perhaps we need something like additional setting for podman to suppress the error.

$ git diff
diff --git a/nextflow.config b/nextflow.config
index 7586d11..fdd3d3e 100644
--- a/nextflow.config
+++ b/nextflow.config
@@ -108,6 +108,7 @@ profiles {
     // Testing this in nf-core after discussion here https://github.com/nf-core/tools/pull/351
     // once this is established and works well, nextflow might implement this behavior as new default.
     docker.runOptions = '-u \$(id -u):\$(id -g)'
+    docker.mountFlags = 'z'
   }
   singularity {
     singularity.enabled = true
@@ -115,6 +116,7 @@ profiles {
   }
   podman      {
     podman.enabled = true
+    podman.mountFlags = 'z'
   }
   test                 { includeConfig 'conf/test.config'                 }
   test_full            { includeConfig 'conf/test_full.config' 

[junaruga]

I find this issue related to a file permission under the mounted directory. You can see containers/podman#2379 . In the page, when root owned file existed, the error relabel failed happened.

On my local environment, the result is like this. The error happens when mounting my home directory.

$ podman run -t -v /home/jaruga:/test:Z docker.io/nfcore/nanoseq uname
Error: relabel failed "/home/jaruga": SELinux relabeling of /home/jaruga is not allowed

The error does not happen when mounting a sub directory.

$ podman run -t -v /home/jaruga/tmp:/test:Z docker.io/nfcore/nanoseq uname
Linux

When checking the following command, I had some file that the owner is not my user account (jaruga) for. I changed the owner to jaruga.

$ find /home/jaruga ! -user jaruga

But after that, the error happens on podman run -t -v /home/jaruga:/test:Z docker.io/nfcore/nanoseq uname. I do not know why.

Why does this nanoseq or nextflow need to mount the home directory /home/jaruga internally?

[junaruga]

If someone is interested in this issue, and you have Debian or Ubuntu (groovy) latest stable version, you can install the distribution podman package and SELinux.

• https://packages.ubuntu.com/groovy/podman
• https://wiki.ubuntu.com/SELinux

[junaruga]

In case of rnaseq

$ git diff
diff --git a/nextflow.config b/nextflow.config
index c8cef83..47e196f 100644
--- a/nextflow.config
+++ b/nextflow.config
@@ -134,6 +134,7 @@ profiles {
     // Testing this in nf-core after discussion here https://github.com/nf-core/tools/pull/351
     // once this is established and works well, nextflow might implement this behavior as new default.
     docker.runOptions = '-u \$(id -u):\$(id -g)'
+    docker.mountFlags = 'z'
   }
   singularity {
     singularity.enabled = true
@@ -141,6 +142,7 @@ profiles {
   }
   podman {
     podman.enabled = true
+    podman.mountFlags = 'z'
   }
   test        { includeConfig 'conf/test.config'      }
   test_sra    { includeConfig 'conf/test_sra.config'  }

$ pwd
/home/jaruga/git/nf-core/rnaseq

$ nextflow run nf-core/rnaseq -profile test,podman 2>&1 | tee nextflow-run-rnaseq-test-podman1.log
...
Error executing process > 'RNASEQ:QUANTIFY_SALMON:UNTAR (salmon.tar.gz)'

Caused by:
  Process `RNASEQ:QUANTIFY_SALMON:UNTAR (salmon.tar.gz)` terminated with an error exit status (126)

Command executed:

  tar -xzvf  salmon.tar.gz
  echo $(tar --version 2>&1) | sed 's/^.*(GNU tar) //; s/ Copyright.*$//' > untar.version.txt

Command exit status:
  126

Command output:
  (empty)

Command error:
  Trying to pull docker.io/biocontainers/biocontainers:v1.2.0_cv1...
  Getting image source signatures
  Copying blob sha256:a4a2a29f9ba48efd3d2075f395538b2eec56fb1bedfb7aecf5e54174446f9e2a
  Copying blob sha256:866d6c39941674c2f103400ef929269b7f5ef97e44aa9fb6d505a60414363013
  Copying blob sha256:4039240d2e0b4bcb42ccbce75bc54570e471ad81457478de35fbeef63536e9c0
  Copying blob sha256:127c9761dcbaa288abc58fc56437c2f2ffbe611b9f7f30e0b5b43cd348bb2094
  Copying blob sha256:cd9d8a5907de2a0b29a08cbb52c00b38216edb81379ee6ed5c6dbdb282fbf2e5
  Copying blob sha256:d13bf203e905463e64d89b14509aafa983fb8baf7c1931fe0a65652aeb6c838f
  Copying blob sha256:eadce7eb40916880bc2e647ee147a0af0e1087c02f128faa2518fc0ab3b67737
  Copying blob sha256:90bae14c26aa8605af58f5473d750448b6ee990435bc778ae0b5a4d949b37fad
  Copying blob sha256:edd3bf038310c1ae88d927a7dcd473c153e9841016eead202e38d0b4facf6776
  Copying blob sha256:926449ce537bc4166f36dd99865ff7ec4ec9e06bb65baa0e0a832d73a56afff0
  Copying blob sha256:52d2b413fa5812d94fda8ec1eb32ca64d5680ad4595e748c69a98a2f3f28ef94
  Copying blob sha256:000c5021555dd0cb32171395ef769a00033f6e8cbc1aaf69283db89bdeea517a
  Copying blob sha256:9ce3805c05ff737dc496106f1a71c25a31f99a95addbfbee6f0f857e8759ff9a
  Copying blob sha256:55217f90a8cafbd66ff1f09a3d006d6b7bcae2dd967b54b420a9d148e8418def
  Copying config sha256:71aff344337cc87284d0a4288e36231519129e24745cbd1d0fabf6771c5a40ea
  Writing manifest to image destination
  Storing signatures
  Error: relabel failed "/home/jaruga": SELinux relabeling of /home/jaruga is not allowed

Work dir:
  /home/jaruga/git/nf-core/rnaseq/work/9c/9a8c87b2e9357bc7dd786842130f17

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

[thyagoleal]

Dear,
I'm having the same issue:
/bin/bash: .command.run: Permission denied

$ getenforce
Disabled

$ sestatus
SELinux status:                 disabled 

The pipeline is on a shared server with non-root privilleges. I'm calling the pipeline from a dir down from my home.

Does anyone know how should I proceed?

Attached is my nextflow log.
.nextflow.log

------------------------ Update 1 ------------------------
I noticed that the .command.run is owned by me, but without execution privilleges (-rw-rw----), which I think is the culprit.

------------------------ Update 2 ------------------------

What I've tried so far :

I'm using only a custom nextflow.conf (via -C) with any of the following changes in the docker settings.

docker.runOptions = '--user 1081:1123'
docker.runOptions = '-u \$(id -u):\$(id -g)'
docker.runOptions = '--user root'
docker.mountFlags = 'z'
docker.fixOwnership = true
docker.userEmulation = false

I'm using the following to run the pipeline (online):
nextflow -C nextflow.config run nf-core/rnaseq -r 3.5 -profile docker

Thank you

@drpatelh

[aubreybailey]

There are a couple big issues with docker/podman, selinux and nextflow.

The first and biggest is using selinux context labels:

Ideally if you're running SELinux you're on RHEL8 (or equivalent) and you can choose podman instead of docker and do this rootlessly. It shouldnt need a mountFlags directive in your config at all if you're doing this.

Understandably you dont want docker content breaking out into your home directory, so this is disallowed.
The fix is to do your analysis in a dedicated root area e.g. /data, and relabel the SELinux contexts in that area so that docker et al. can edit their own labels.

Here's the workaround:
DONT do this in your home directory, but you can relabel a dedicated root area (assuming /podman_analysis/) for container label changes (i.e. the Z or z flags) using:

restorecon -R -v -F /podman_analysis/
chcon -Rt container_file_t /podman_analysis/

It's stupid, but just doing your analysis in a dedicated directory fixes the relabeling problems

The second big problem is that modern kernels support resource limits via cgroups v2, and it will give a weird 'acccess denied' usually with some cryptic hint about 'not being allowed to set a memory limit.'
The workaround here is to install crun and change the containers runtime to run as crun instead of runc (this is faster anyways!)

dnf install crun
cp  /usr/share/containers/containers.conf /etc/containers/
nano  /etc/containers/containers.conf 
   > cgroups="disabled"
   > runtime="crun"