[bug] RuntimeError from Gubbins via Singularity in Snippy subworkflow

[pcrxn]

Description
When running the --wf snippy subworkflow with -profile singularity, a RuntimeError is raised by Gubbins within the BACTOPIATOOLS:SNIPPY:GUBBINS:GUBBINS_MODULE (snippy-core) process.

Steps to Reproduce
Steps to reproduce the behavior:

nextflow run bactopia/bactopia \
    -r v3.0.1 \
    -profile singularity \
    -resume \
    --wf snippy \
    --reference data/reference/gbff/GCF_002039585.1.gbff \
    --bactopia bactopia/assembly

[...]

[e3/7a4489] NOTE: Process `BACTOPIATOOLS:SNIPPY:GUBBINS:GUBBINS_MODULE (snippy-core)` terminated with an error exit status (1) -- Execution is retried (1)
[33/727e0d] NOTE: Process `BACTOPIATOOLS:SNIPPY:GUBBINS:GUBBINS_MODULE (snippy-core)` terminated with an error exit status (1) -- Execution is retried (2)
[5b/d1ec18] NOTE: Process `BACTOPIATOOLS:SNIPPY:GUBBINS:GUBBINS_MODULE (snippy-core)` terminated with an error exit status (1) -- Execution is retried (3)
ERROR ~ Error executing process > 'BACTOPIATOOLS:SNIPPY:GUBBINS:GUBBINS_MODULE (snippy-core)'

Caused by:
  Process `BACTOPIATOOLS:SNIPPY:GUBBINS:GUBBINS_MODULE (snippy-core)` terminated with an error exit status (1)

Command executed:

  if [ "true" == "true" ]; then
      gzip -c -d core-snp-clean.full.aln.gz > core-snp-clean.full.aln
  fi
  
  run_gubbins.py \
      --threads 4 \
      --prefix core-snp \
      --iterations 5 --min-snps 3 --min-window-size 100 --max-window-size 10000 --filter-percentage 25.0 \
      core-snp-clean.full.aln
  
  # Create masked alignment
  mask_gubbins_aln.py \
      --aln core-snp-clean.full.aln \
      --gff core-snp.recombination_predictions.gff \
      --out core-snp.masked.aln
  
  # Cleanup
  gzip *.masked.aln *.embl *.fasta *.gff *.vcf
  
  cat <<-END_VERSIONS > versions.yml
  "BACTOPIATOOLS:SNIPPY:GUBBINS:GUBBINS_MODULE":
      gubbins: $(run_gubbins.py --version 2>&1)
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Converting SIF file to temporary sandbox...
  Traceback (most recent call last):
    File "/usr/local/bin/run_gubbins.py", line 7, in <module>
      from gubbins.run_gubbins import main
    File "/usr/local/lib/python3.9/site-packages/gubbins/__init__.py", line 16, in <module>
      from gubbins import common
    File "/usr/local/lib/python3.9/site-packages/gubbins/common.py", line 42, in <module>
      from gubbins.pyjar import jar, get_base_patterns
    File "/usr/local/lib/python3.9/site-packages/gubbins/pyjar.py", line 212, in <module>
      def seq_to_int(seq,out_seq):
    File "/usr/local/lib/python3.9/site-packages/numba/core/decorators.py", line 229, in wrapper
      disp.enable_caching()
    File "/usr/local/lib/python3.9/site-packages/numba/core/dispatcher.py", line 856, in enable_caching
      self._cache = FunctionCache(self.py_func)
    File "/usr/local/lib/python3.9/site-packages/numba/core/caching.py", line 601, in __init__
      self._impl = self._impl_class(py_func)
    File "/usr/local/lib/python3.9/site-packages/numba/core/caching.py", line 337, in __init__
      raise RuntimeError("cannot cache function %r: no locator available "
  RuntimeError: cannot cache function 'seq_to_int': no locator available for file '/usr/local/lib/python3.9/site-packages/gubbins/pyjar.py'
  INFO:    Cleaning up image...

Expected Behavior

The BACTOPIATOOLS:SNIPPY:GUBBINS:GUBBINS_MODULE (snippy-core) process should complete successfully.

Execution Environment

• Bactopia Version: v3.0.1
• OS: Linux Mint 21.2
• Environment: Singularity on local computer

Additional Information

Nextflow execution log attached.

[rpetit3]

Hi @pcrxn

I was wondering if someone would run into this. I ran into it with CI testing last weekend, and I think its related to Nextflow no longer mounting home directories with Singularity, and Numba trying to write something to the home directory.

From nextflow docs:

Changed in version 23.07.0-edge: Nextflow no longer mounts the home directory when launching a Singularity container. To re-enable the old behavior, set the environment variable NXF_SINGULARITY_HOME_MOUNT to true.

Can you try exporting the following variable?

export NXF_SINGULARITY_HOME_MOUNT=true

[pcrxn]

Hi @rpetit3,

Thanks for the quick response. Running export NXF_SINGULARITY_HOME_MOUNT=true and then running the subworkflow again worked! New log attached.

[rpetit3]

Nice to know!

I will look into this to see if there is a way to avoid the usage NXF_SINGULARITY_HOME_MOUNT. I think only gubbins via Singularity was affected by this.

[rpetit3]

For future use cases with Apptainer

export NXF_APPTAINER_HOME_MOUNT=true