Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

clean_work_dirs bug #263

Open
cbmckni opened this issue Apr 22, 2022 · 3 comments
Open

clean_work_dirs bug #263

cbmckni opened this issue Apr 22, 2022 · 3 comments
Labels
bug

Comments

@cbmckni
Copy link

cbmckni commented Apr 22, 2022

Description of the bug

I am running GEMmaker on the PRP with a human kidney dataset stored locally in split fastq format. Just with the CORG reference for now to test.

Command used:

nextflow -C nextflow.config kuberun systemsgenetics/gemmaker -profile k8s -v pvc-storage-test --pipeline kallisto --kallisto_index_path /workspace/projects/systemsgenetics/gemmaker/assets/demo/references/CORG.transcripts.Kallisto.indexed --input /workspace/data/kidneydata/sra/*.fastq --outdir /workspace/gemmaker/output --max_cpus 8

nextflow.config to add k8s profile:

profiles {
    k8s {
        process {
            executor = "k8s"
        }
        executor {
            queueSize = 16
        }
    }
}

I get the following error, even after deleting the work dir:

[9f/c1136c] Submitted process > GEMmaker:clean_work_dirs (SRR5139395_2)
[3c/9e3485] Submitted process > GEMmaker:fastqc_1 (SRR5139414_2)
Error executing process > 'GEMmaker:clean_work_dirs (SRR5139395_2)'

Caused by:
  Process `GEMmaker:clean_work_dirs (SRR5139395_2)` terminated with an error exit status (1)

Command executed:

  for dir in /workspace/cole/work/24/1ec0308597286e9e24326f8d85c546/SRR5139395_2.Kallisto.ga; do
    if [ -e $dir ]; then
      echo "Cleaning: $dir"
      files=`find $dir -type  f `
  
      echo "Files to delete: $files"
      clean_work_files.sh "$files" "null"
    fi
  done

Command exit status:
  1

Command output:
  Cleaning: /workspace/cole/work/24/1ec0308597286e9e24326f8d85c546/SRR5139395_2.Kallisto.ga
  Files to delete: /workspace/cole/work/24/1ec0308597286e9e24326f8d85c546/SRR5139395_2.Kallisto.ga/run_info.json
  /workspace/cole/work/24/1ec0308597286e9e24326f8d85c546/SRR5139395_2.Kallisto.ga/abundance.h5
  /workspace/cole/work/24/1ec0308597286e9e24326f8d85c546/SRR5139395_2.Kallisto.ga/abundance.tsv
  cleaning 
  cleaning 
  cleaning 

Command error:
  ps: error while loading shared libraries: libprocps.so.4: cannot open shared object file: No such file or directory
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 15: perl: command not found
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 26: $file: ambiguous redirect
  truncate: option requires an argument -- 's'
  Try 'truncate --help' for more information.
  touch: invalid date format ‘@’
  touch: invalid date format ‘@’
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 15: perl: command not found
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 26: $file: ambiguous redirect
  truncate: option requires an argument -- 's'
  Try 'truncate --help' for more information.
  touch: invalid date format ‘@’
  touch: invalid date format ‘@’
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 15: perl: command not found
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 26: $file: ambiguous redirect
  truncate: option requires an argument -- 's'
  Try 'truncate --help' for more information.
  touch: invalid date format ‘@’
  touch: invalid date format ‘@’

Work dir:
  /workspace/cole/work/9f/c1136c3f9d20f7a30ccc0c1c813c2a

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`


WARN: Killing pending tasks (14)

Any idea what the issue may be? I have used a slight variation of this command in the past during the scidas workshop and had no issues.

Command used and terminal output

[cole@localhost gm]$ ./nextflow -C nextflow.config kuberun systemsgenetics/gemmaker    -profile k8s    -v pvc-storage-test   --pipeline kallisto   --kallisto_index_path /workspace/projects/systemsgenetics/gemmaker/assets/demo/references/CORG.transcripts.Kallisto.indexed   --input  /workspace/data/kidneydata/sra/*.fastq   --outdir /workspace/gemmaker/output   --max_cpus 8
Pod started: jovial-spence
N E X T F L O W  ~  version 21.10.6
Launching `systemsgenetics/gemmaker` [jovial-spence] - revision: 9a880fd991 [master]


------------------------------------------------------
                                        ,--./,-.
        ___     __   __   __   ___     /,-._.--~'
  |\ | |__  __ /  ` /  \ |__) |__         }  {
  | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                        `._,._,'
  systemsgenetics/gemmaker v2.1.0
------------------------------------------------------
Core Nextflow options
  revision                  : master
  runName                   : jovial-spence
  launchDir                 : /workspace/cole
  workDir                   : /workspace/cole/work
  projectDir                : /workspace/projects/systemsgenetics/gemmaker
  userName                  : root
  profile                   : standard
  configFiles               : /workspace/projects/systemsgenetics/gemmaker/nextflow.config, /workspace/cole/nextflow.config

Input/output options
  input                     : /workspace/data/kidneydata/sra/*.fastq
  sras                      : null
  outdir                    : /workspace/gemmaker/output
  failed_run_report_template: /workspace/projects/systemsgenetics/gemmaker/assets/failed_sra_runs.template.html
  multiqc_config_file       : /workspace/projects/systemsgenetics/gemmaker/assets/multiqc_config.yaml
  multiqc_custom_logo       : /workspace/projects/systemsgenetics/gemmaker/assets/systemsgenetics-gemmaker_logo.png

Kallisto Pipeline
  kallisto_index_path       : /workspace/projects/systemsgenetics/gemmaker/assets/demo/references/CORG.transcripts.Kallisto.indexed
  kallisto_bootstrap_samples: 0

Reference genome options
  igenomes_ignore           : true

Generic options
  publish_dir_mode          : link

Max job request options
  max_cpus                  : 8

!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use systemsgenetics/gemmaker for your analysis please cite:

* The pipeline
  https://doi.org/10.5281/zenodo.3620945

* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x

* Software dependencies
  https://github.com/systemsgenetics/gemmaker/blob/master/CITATIONS.md
------------------------------------------------------
[99/1d5196] Submitted process > GEMmaker:fastqc_1 (SRR5139394_1)
[7b/1b263a] Submitted process > GEMmaker:fastqc_1 (SRR5139394_2)
[ac/2772e0] Submitted process > GEMmaker:fastqc_1 (SRR5139395_1)
[1d/aeb630] Submitted process > GEMmaker:fastqc_1 (SRR5139395_2)
[ea/1455e0] Submitted process > GEMmaker:fastqc_1 (SRR5139396_1)
[2c/b4ef50] Submitted process > GEMmaker:kallisto (SRR5139394_1)
[9b/d39dd1] Submitted process > GEMmaker:fastqc_1 (SRR5139396_2)
[f3/349dee] Submitted process > GEMmaker:kallisto (SRR5139394_2)
[09/eebb63] Submitted process > GEMmaker:kallisto (SRR5139395_1)
[94/0f2d08] Submitted process > GEMmaker:fastqc_1 (SRR5139397_1)
[24/1ec030] Submitted process > GEMmaker:kallisto (SRR5139395_2)
[c7/bd996a] Submitted process > GEMmaker:fastqc_1 (SRR5139397_2)
[b4/d19a17] Submitted process > GEMmaker:kallisto (SRR5139396_1)
[7e/719739] Submitted process > GEMmaker:kallisto (SRR5139396_2)
[3f/311d73] Submitted process > GEMmaker:kallisto (SRR5139397_1)
[02/a66b26] Submitted process > GEMmaker:kallisto (SRR5139397_2)
[75/f12ea4] Submitted process > GEMmaker:kallisto_tpm (SRR5139395_2)
[2b/5a9a55] Submitted process > GEMmaker:next_sample (SRR5139395_2)
[9f/c1136c] Submitted process > GEMmaker:clean_work_dirs (SRR5139395_2)
[3c/9e3485] Submitted process > GEMmaker:fastqc_1 (SRR5139414_2)
Error executing process > 'GEMmaker:clean_work_dirs (SRR5139395_2)'

Caused by:
  Process `GEMmaker:clean_work_dirs (SRR5139395_2)` terminated with an error exit status (1)

Command executed:

  for dir in /workspace/cole/work/24/1ec0308597286e9e24326f8d85c546/SRR5139395_2.Kallisto.ga; do
    if [ -e $dir ]; then
      echo "Cleaning: $dir"
      files=`find $dir -type  f `
  
      echo "Files to delete: $files"
      clean_work_files.sh "$files" "null"
    fi
  done

Command exit status:
  1

Command output:
  Cleaning: /workspace/cole/work/24/1ec0308597286e9e24326f8d85c546/SRR5139395_2.Kallisto.ga
  Files to delete: /workspace/cole/work/24/1ec0308597286e9e24326f8d85c546/SRR5139395_2.Kallisto.ga/run_info.json
  /workspace/cole/work/24/1ec0308597286e9e24326f8d85c546/SRR5139395_2.Kallisto.ga/abundance.h5
  /workspace/cole/work/24/1ec0308597286e9e24326f8d85c546/SRR5139395_2.Kallisto.ga/abundance.tsv
  cleaning 
  cleaning 
  cleaning 

Command error:
  ps: error while loading shared libraries: libprocps.so.4: cannot open shared object file: No such file or directory
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 15: perl: command not found
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 26: $file: ambiguous redirect
  truncate: option requires an argument -- 's'
  Try 'truncate --help' for more information.
  touch: invalid date format ‘@’
  touch: invalid date format ‘@’
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 15: perl: command not found
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 26: $file: ambiguous redirect
  truncate: option requires an argument -- 's'
  Try 'truncate --help' for more information.
  touch: invalid date format ‘@’
  touch: invalid date format ‘@’
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 15: perl: command not found
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  stat: missing operand
  Try 'stat --help' for more information.
  /workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 26: $file: ambiguous redirect
  truncate: option requires an argument -- 's'
  Try 'truncate --help' for more information.
  touch: invalid date format ‘@’
  touch: invalid date format ‘@’

Work dir:
  /workspace/cole/work/9f/c1136c3f9d20f7a30ccc0c1c813c2a

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`


WARN: Killing pending tasks (14)
[26/029fea] Submitted process > GEMmaker:kallisto (SRR5139414_2)
WARN: To render the execution DAG in the required format it is required to install Graphviz -- See http://www.graphviz.org for more info.



### Relevant files

_No response_

### System information

nextflow version 21.10.6.5660

K8s executor

systemsgenetics/gemmaker v2.1.0
@cbmckni cbmckni added the bug label Apr 22, 2022
@cbmckni
Copy link
Author

cbmckni commented Apr 22, 2022

I get the same error running with the locally stored CORG fastq.gz files, so that should be easier to replicate:

./nextflow -C nextflow.config kuberun systemsgenetics/gemmaker    -profile k8s    -v pvc-storage-test   --pipeline kallisto   --kallisto_index_path /workspace/projects/systemsgenetics/gemmaker/assets/demo/references/CORG.transcripts.Kallisto.indexed   --input  /workspace/projects/systemsgenetics/gemmaker/assets/demo/*.fastq.gz   --outdir /workspace/gemmaker/output   --max_cpus 4

@spficklin
Copy link
Member

spficklin commented Apr 24, 2022 •
edited

It seems the problem is indicated with these lines in the error log:

First:

ps: error while loading shared libraries: libprocps.so.4: cannot open shared object file: No such file or directory

Followed by:

/workspace/projects/systemsgenetics/gemmaker/bin/clean_work_files.sh: line 15: perl: command not found

Then by:

stat: missing operand

I'm not sure where the ps error is coming from because the bin/clean_work_files.sh script which is being called doesn't use ps. But the perl error is occurring at this line:

GEMmaker/bin/clean_work_files.sh

Line 15 in 9a880fd

file=`echo $file | perl -p -e 's/[\\[,\\]]//g'`

The clean_work_dirs process doesn't use a container. It's just basic BASH code and since we figured perl was ubiquitous we didn't bother to create a container for this step. So, my thinking is that whatever container is being used on your k8s system is missing perl, libraries for ps and a different stat command.

@spficklin
Copy link
Member

Perhaps, the best solution would probably be to specify a container that this step should run in, that has the correct libraries, and binaries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@spficklin @cbmckni