Unable to use publishDir to publish a large directory

[sbalgobin94]

I am using publishDir to copy an output directory (that has subdirectories) from my Cellranger process to a specified directory. For smaller output directories, this works, but not for a larger one (>50G total). I am using the Slurm executor and using a shared file system.

What I've tried so far:

I have cleared the cache
Changed my OpenJDK to Corretto (it was previously JetBrains)
Tried another file system
Tried other runs of the same size from the same directory
Tried smaller runs (successful)
Tried to gzip the output folder, doesn't help

Here is my nextflow script:

params.run_dir = ''
params.sample_id = ''
params.csv = ''
params.workflow = ''
params.gex_outdir = '/work/srb108/GEX'


process makefastq_gex {
    cache 'lenient'
    beforeScript 'module load Cell-Ranger/7.2.0'
    publishDir "${params.gex_outdir}", mode: 'copy'

    input:
    path run_dir
    val sample_id
    path csv
    
    output:
    path "${sample_id}_mkfastq_outs"

    script:
    """
    cellranger mkfastq --run=${run_dir} --csv=${csv} --output-dir=${sample_id}_mkfastq_outs --delete-undetermined
    """

} 


workflow {
    run_dir = Channel.fromPath(params.run_dir)
    sample_id = params.sample_id
    csv = Channel.fromPath(params.csv)
    workflow = params.workflow

    if(params.workflow == "gex") {
        makefastq_gex(run_dir, sample_id, csv)
    }  
    
}

Here is my config file:

workDir = '/work/srb108'

process {
    executor = 'slurm'

    withName: makefastq_gex {
        cpus = 10
        memory = '250 GB'
        clusterOptions = '--partition=dhvi --job-name=test_mkfastq_job --mail-type=END --mail-user=srb108@duke.edu'
    }

}

This is the ending portion of a log file for a run in which publishDir was SUCCESSFUL:

Mar-18 14:52:11.105 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=10; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Mar-18 14:52:11.254 [main] DEBUG nextflow.Session - Session await > all processes finished
Mar-18 14:52:11.280 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm) - terminating tasks monitor poll loop
Mar-18 14:52:11.285 [main] DEBUG nextflow.Session - Session await > all barriers passed
Mar-18 14:52:16.328 [main] INFO  nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (1 files)
^ THIS LINE REPEATS A FEW TIMES
Mar-18 14:58:42.520 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Mar-18 14:58:43.105 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=7h 41m 42s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=10; peakMemory=100 GB; ]
Mar-18 14:58:43.731 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Mar-18 14:58:44.215 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Mar-18 14:58:44.216 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

However, I do not see this ending portion for the runs whose results do not get published as indicated in this log file. These are the last few lines and that's it:

Mar-25 16:02:02.696 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 5508033; id: 1; name: makefastq_gex (1); status: RUNNING; exit: -; error: -; workDir: /work/srb108/3e/ec9f059261a38bd6ff58c6ced0911b started: 1711387919417; exited: -; ]
Mar-25 16:07:02.702 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 5508033; id: 1; name: makefastq_gex (1); status: RUNNING; exit: -; error: -; workDir: /work/srb108/3e/ec9f059261a38bd6ff58c6ced0911b started: 1711387919417; exited: -; ]
Mar-25 16:12:02.709 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below
~> TaskHandler[jobId: 5508033; id: 1; name: makefastq_gex (1); status: RUNNING; exit: -; error: -; workDir: /work/srb108/3e/ec9f059261a38bd6ff58c6ced0911b started: 1711387919417; exited: -; ]

There is nothing in .command.err.

It looks like the publishDir thread pool is not being created for these larger runs. I'm at a loss for where to go from here. Even if I used saveAs, there are different files in each subdirectory and I want to keep things organized. I want to keep the entire folder from the Cellranger process so that the user can navigate to it whenever they need to.

[bentsherman]

Hi Samantha, a few probing questions:

• what versions of Nextflow and Corretto Java?
• does the large run crash or hang? is it killed by the scheduler? how does it terminate?
• can you give me a listing of the directory structure (e.g. ls -R or find) so that I can see what it looks like?

[sbalgobin94]

Hi @bentsherman thanks for helping!

• Nextflow version 23.10.1.5891
• Corretto-17.0.6.10.1
• I'm not sure what hang means, but it definitely does not crash. The scheduler says that the job is completed. It terminates with a 0. In the log file, it seems like publishDir is never called, it appears as if the script is still running even though it's ended. This is the last line in the nextflow.log file: Mar-25 16:12:02.709 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor slurm > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[jobId: 5508033; id: 1; name: makefastq_gex (1); status: RUNNING; exit: -; error: -; workDir: /work/srb108/3e/ec9f059261a38bd6ff58c6ced0911b started: 1711387919417; exited: -; ]
• This is the Slurm notification:
  Slurm Job_id=5635522 Name=test_mkfastq_job Ended, Run time 07:09:26, COMPLETED, ExitCode 0
• This is the the directory structure using ls -R
  `P018_mkfastq_outs:
  HNG5LDSX3 Reports Stats

P018_mkfastq_outs/HNG5LDSX3:
P018_1_S1_L001_I1_001.fastq.gz P018_1_S1_L002_I1_001.fastq.gz P018_1_S1_L003_I1_001.fastq.gz P018_1_S1_L004_I1_001.fastq.gz
P018_1_S1_L001_I2_001.fastq.gz P018_1_S1_L002_I2_001.fastq.gz P018_1_S1_L003_I2_001.fastq.gz P018_1_S1_L004_I2_001.fastq.gz
P018_1_S1_L001_R1_001.fastq.gz P018_1_S1_L002_R1_001.fastq.gz P018_1_S1_L003_R1_001.fastq.gz P018_1_S1_L004_R1_001.fastq.gz
P018_1_S1_L001_R2_001.fastq.gz P018_1_S1_L002_R2_001.fastq.gz P018_1_S1_L003_R2_001.fastq.gz P018_1_S1_L004_R2_001.fastq.gz

P018_mkfastq_outs/Reports:
html

P018_mkfastq_outs/Reports/html:
HNG5LDSX3 index.html Report.css tree.html

P018_mkfastq_outs/Reports/html/HNG5LDSX3:
all default HNG5LDSX3

P018_mkfastq_outs/Reports/html/HNG5LDSX3/all:
all

P018_mkfastq_outs/Reports/html/HNG5LDSX3/all/all:
all unknown

P018_mkfastq_outs/Reports/html/HNG5LDSX3/all/all/all:
laneBarcode.html lane.html

P018_mkfastq_outs/Reports/html/HNG5LDSX3/all/all/unknown:

P018_mkfastq_outs/Reports/html/HNG5LDSX3/default:
all Undetermined

P018_mkfastq_outs/Reports/html/HNG5LDSX3/default/all:
all unknown

P018_mkfastq_outs/Reports/html/HNG5LDSX3/default/all/all:
laneBarcode.html lane.html

P018_mkfastq_outs/Reports/html/HNG5LDSX3/default/all/unknown:

P018_mkfastq_outs/Reports/html/HNG5LDSX3/default/Undetermined:
all unknown

P018_mkfastq_outs/Reports/html/HNG5LDSX3/default/Undetermined/all:
laneBarcode.html lane.html

P018_mkfastq_outs/Reports/html/HNG5LDSX3/default/Undetermined/unknown:
laneBarcode.html lane.html

P018_mkfastq_outs/Reports/html/HNG5LDSX3/HNG5LDSX3:
all P018_1

P018_mkfastq_outs/Reports/html/HNG5LDSX3/HNG5LDSX3/all:
all unknown

P018_mkfastq_outs/Reports/html/HNG5LDSX3/HNG5LDSX3/all/all:
laneBarcode.html lane.html

P018_mkfastq_outs/Reports/html/HNG5LDSX3/HNG5LDSX3/all/unknown:

P018_mkfastq_outs/Reports/html/HNG5LDSX3/HNG5LDSX3/P018_1:
all GTAGCCCTGT+ATAGATGCTC unknown

P018_mkfastq_outs/Reports/html/HNG5LDSX3/HNG5LDSX3/P018_1/all:
laneBarcode.html lane.html

P018_mkfastq_outs/Reports/html/HNG5LDSX3/HNG5LDSX3/P018_1/GTAGCCCTGT+ATAGATGCTC:
laneBarcode.html lane.html

P018_mkfastq_outs/Reports/html/HNG5LDSX3/HNG5LDSX3/P018_1/unknown:

P018_mkfastq_outs/Stats:
AdapterTrimming.txt DemultiplexingStats.xml DemuxSummaryF1L2.txt DemuxSummaryF1L4.txt FastqSummaryF1L2.txt FastqSummaryF1L4.txt
ConversionStats.xml DemuxSummaryF1L1.txt DemuxSummaryF1L3.txt FastqSummaryF1L1.txt FastqSummaryF1L3.txt Stats.json`

[bentsherman]

"hang" just means that the process runs forever. but it looks like your job simply "completed" with exit code 0

The log for the unsuccessful run looks truncated, as if Nextflow was abruptly terminated. If Nextflow received a normal cancellation signal you would at least see some cleanup messages.

If slurm killed the process I would expect to see some message to that effect, but I don't know how else to explain it. The fact that your workflow runs fine with small data, but mysteriously terminates prematurely with large data, tells me that the HPC system is killing it for using too much resources.

I assume you are submitting the nextflow run as a slurm job which submits other slurm jobs? If you run Nextflow on the head node it will almost certainly be killed on large runs

We can at least try to narrow down the root cause. Try publishing with symlink mode instead of copy. It will make the publish operation very cheap, and should tell us whether the problem is caused by the publishing or something else early on.