Custom config being ignored

[sklages]

Description of the bug

• Custom config file:

process {
    withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR' {
        // single job
        memory = 80.GB
        cpus   = 32
        time   = 4d
    },
    withName: 'FASTQ_SUBSAMPLE_FQ_SALMON:SALMON_INDEX' {
        // salmon index --threads 6 <..>
        memory = 90.GB
        cpus   = 24
        time   = 3d
    }
}

These numbers/resources have been chosen for better identification/discrimination in log files.

When running the workflow, I always get a warning when providing the above custom config file:

WARN: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Multiple config files detected!
  Please provide pipeline parameters via the CLI or Nextflow '-params-file' option.
  Custom config files including those provided by the '-c' Nextflow option can be
  used to provide any configuration except for parameters.

  Docs: https://nf-co.re/usage/configuration#custom-configuration-files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This results of the custom config file being ignored. During the run it can be checked (e.g. in top I see salmon index --threads 6 <..>)

• Parameter overview just before starting shows that configFiles is empty:

<..>
N E X T F L O W  ~  version 23.10.0
<..>
Core Nextflow options
  revision             : 3.14.0
  runName              : agitated_ramanujan
  containerEngine      : singularity
  launchDir            : /path/to/nf-rna
  workDir              : /path/to/nf-rna/work
  projectDir           : /path/to/nf-rna/rnaseq/nxf_home/assets/nf-core/rnaseq
  userName             : $USER
  profile              : singularity
  configFiles          :

Final execution_report_2024-03-15_10-37-30.html reports for STAR:

NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN 
  "allocated cpus" -> 12
  "allocated memory" -> 72 GB
  "allocated time" -> 16h 0m

.. and these are the defaults loaded from conf/base.config for process_high labelled processes.

I am aware, that this is probably something very "special" since I couldn't find any "Issue" dealing with that problem.

Command used and terminal output

nextflow run nf-core/rnaseq \
  -config "rnaseq.config" \
  -revision 3.14.0 \
  -profile singularity \
  --max_memory $MAX_MEM \
  --max_cpus $MAX_CPU \
  --max_time $MAX_TIME \
  --input  $CSV \
  --outdir $OUTDIR \
  --star_index $GENOME_DIR \
  --gtf $GENOME_GTF \
  --fasta $GENOME_FSA \
  --igenomes_ignore \
  --genome null \
  --gencode

Relevant files

nexflow.log

System information

Nextflow version 23.10.0 on a dedicated server with local storage (128 cpus/ 1TB), inhouse-made Linux, "local" executor, Singularity

[jdcla]

I am struggling with the same problem.

[jdcla]

Actually, I seemed to just have solved it.

I think nextflow is ignoring config parameters in case these are formatted incorrectly.

Try quoting your strings

process {
    withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR' {
        // single job
        memory = '80.GB'
        cpus   = 32
        time   = '4d'
    },
    withName: 'FASTQ_SUBSAMPLE_FQ_SALMON:SALMON_INDEX' {
        // salmon index --threads 6 <..>
        memory = '90.GB'
        cpus   = 24
        time   = '3d'
    }
}

[sklages]

@jdcla - no difference here .. salmon index is started with --threads 6 (as shown by top), and STAR with --runThreadN 12, both defaults read from conf/base.config

[jdcla]

Actually, you probably also have to separate the properties of the two processes.

process {
    withName: 'NFCORE_RNASEQ:RNASEQ:ALIGN_STAR' {
        // single job
        memory = '80.GB'
        cpus   = 32
        time   = '4d'
    }
}
process {
    withName: 'FASTQ_SUBSAMPLE_FQ_SALMON:SALMON_INDEX' {
        // salmon index --threads 6 <..>
        memory = '90.GB'
        cpus   = 24
        time   = '3d'
    }
}

[pinin4fjords]

The process selector was incorrect in the OP. You should have specified:

NFCORE_RNASEQ:RNASEQ:ALIGN_STAR:STAR_ALIGN

while you specified

NFCORE_RNASEQ:RNASEQ:ALIGN_STAR

(align_star is just the subworkflow in this case, so the selector wasn't hitting the process).

Note also that this is incorrect:

FASTQ_SUBSAMPLE_FQ_SALMON:SALMON_INDEX

... since it's incomplete. You can use the process name only (SALMON_INDEX), the fully qualified process name (including the workflow part), or use a wildcard:

.*:FASTQ_SUBSAMPLE_FQ_SALMON:SALMON_INDEX

Closing the issue now, feel free to reopen if changing as above doesn't help.