Not able to modify fastq_merge memory default

[JohnHadish]

Description of the bug

Upon restart, GEMmaker will run for around 20 minutes then throw the following message. The main node which launches all jobs continues to run without launching any new jobs. The output log file immediatly ends without throwing an error or specifying a direcory where GEMmaker failed.

The error file looks like this:

Exception in thread "Task submitter" groovy.lang.MissingMethodException: No signature of method: nextflow.script.ScriptBinding.check_max() is applicable for argument types: () values: []
	at nextflow.script.WorkflowBinding.invokeMethod(WorkflowBinding.groovy:102)
	at groovy.lang.GroovyObject$invokeMethod$1.call(Unknown Source)
	at nextflow.script.BaseScript.invokeMethod(BaseScript.groovy:156)
	at groovy.lang.GroovyObject$invokeMethod$1.call(Unknown Source)
	at nextflow.processor.TaskContext.invokeMethod(TaskContext.groovy:152)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeOnDelegationObjects(ClosureMetaClass.java:408)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:350)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:61)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:194)
	at _nf_config_f8a1a526$_run_closure1$_closure2$_closure3$_closure8$_closure11.doCall(_nf_config_f8a1a526:21)
	at _nf_config_f8a1a526$_run_closure1$_closure2$_closure3$_closure8$_closure11.doCall(_nf_config_f8a1a526)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.lang.Closure.call(Closure.java:412)
	at groovy.lang.Closure.call(Closure.java:406)
	at nextflow.processor.LazyMap.resolveImpl(TaskConfig.groovy:572)
	at nextflow.processor.LazyMap.resolveImpl(TaskConfig.groovy)
	at nextflow.processor.LazyMap.resolve(TaskConfig.groovy:533)
	at nextflow.processor.LazyMap.getValue(TaskConfig.groovy:600)
	at nextflow.processor.LazyMap.get(TaskConfig.groovy:595)
	at nextflow.processor.TaskConfig.get(TaskConfig.groovy:146)
	at nextflow.processor.TaskConfig.getMemory(TaskConfig.groovy:223)
	at nextflow.processor.TaskConfig$getMemory$6.call(Unknown Source)
	at nextflow.processor.TaskHandler.getTraceRecord(TaskHandler.groovy:169)
	at nextflow.executor.GridTaskHandler.super$2$getTraceRecord(GridTaskHandler.groovy)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1268)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:144)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:164)
	at nextflow.executor.GridTaskHandler.getTraceRecord(GridTaskHandler.groovy:394)
	at nextflow.Session.notifyTaskComplete(Session.groovy:972)
	at nextflow.processor.TaskPollingMonitor.submitPendingTasks(TaskPollingMonitor.groovy:566)
	at nextflow.processor.TaskPollingMonitor.submitLoop(TaskPollingMonitor.groovy:387)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1268)
	at groovy.lang.MetaClassImpl.invokeMethodClosure(MetaClassImpl.java:1048)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1142)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.lang.Closure.call(Closure.java:412)
	at groovy.lang.Closure.call(Closure.java:406)
	at groovy.lang.Closure.run(Closure.java:493)
	at java.lang.Thread.run(Thread.java:745)

Command used and terminal output

No response

Relevant files

No response

System information

No response

[spficklin]

Hey @JohnHadish can you provide more info. What version of nextflow, java, etc, what mode, what parameters, etc.

[JohnHadish]

Alright, I think I may have fixed it. The issue appears to be that GEMmaker does not like how I specified memory for fastq_merge processes. The workflow was running as expected, but failed due to running out of memory. After this I changed the fastq_merge parameter in the nextflow.config to permit more memory usage:

	withName: fastq_merge {
          memory = { check_max( 6.GB * task.attempt, 'memory' ) }
        }

My entire nextflow.config then looked like this:

profiles {
   kamiak {
     process {
        executor = "slurm"
	errorStrategy = "retry"        
	maxRetries = 3

	withName: retrieve_sra_metadata {
          memory = 32.GB
	}
	
	withName: create_gem {
	  memory = { check_max( 8.GB * task.attempt, 'memory' ) }
	}
	
	withName: multiqc {
          memory = { check_max( 8.GB * task.attempt, 'memory' ) }
        }

      	withName: fastq_merge {
          memory = { check_max( 6.GB * task.attempt, 'memory' ) }
        }

     }
     executor {
       queueSize = 120
     }
  }
}

I am not sure why this would have produced the above error, as I have specified other memory requirments in the same way. I can not find anything in the workflow which would make fastq_merge fall under different specifications.

Upon removing the new memory requirments, it appears that everything is running.

nextflow/21.10.6
java/1.8.0
singularity/3.8.0

[JohnHadish]

Replicate using the test data:
Command:

projectDir="/home/john.hadish/.nextflow/assets/systemsgenetics/gemmaker"

nextflow run systemsgenetics/gemmaker -r gem_fix \
  -profile kamiak,singularity \
  -resume \
  --pipeline kallisto \
  --sras "${projectDir}/assets/demo/SRA_IDs.txt" \
  --input "${projectDir}/assets/demo/*{1,2}.fastq" \
  --skip_samples "${projectDir}/assets/demo/samples2skip.txt" \
  --kallisto_index_path "${projectDir}/assets/demo/references/CORG.transcripts.Kallisto.indexed" \
  --max_cpus 80

nextflow.config:

profiles {
   kamiak {
     process {
        executor = "slurm"
        queue = "ficklin"
        clusterOptions = "--account=ficklin"
	errorStrategy = "retry"        
	maxRetries = 5

	withName: retrieve_sra_metadata {
          memory = 32.GB
	}
	
	withName: create_gem {
          memory = { check_max( 8.GB * task.attempt, 'memory' ) }
        }

        withName: multiqc {
          memory = { check_max( 8.GB * task.attempt, 'memory' ) }
        }

	withName: fastq_merge {
          memory = { check_max( 6.GB * task.attempt, 'memory' ) }
        }


     }
     executor {
       queueSize = 90
     }
  }
}