Unable to run nextpolish program

[amit4mchiba]

Hi, I am writing here to request your help to run Nextpolish. I have long read (pacbio sequel 1) and illumina reads, and I wanted to polish the assembly. After installing the software, I was able to run the test, which means the installation has no problem.

Next, based on test, I created config file and run the program, and got following error-
[154451 INFO] 2022-03-02 02:47:30 Submitted jobID:[154452] jobCmd:[/mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir3/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh] in the local_cycle.
[154452 CRITICAL] 2022-03-02 02:47:30 Command '/bin/sh /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir3/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh > /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir3/00.lgs_polish/01.db_split.sh.work/db
_split1/nextPolish.sh.o 2> /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir3/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh.e' returned non-zero exit status 1, error info: .
Traceback (most recent call last):
File "/mnt/HD1/NextPolish/nextPolish", line 515, in
main(args)
File "/mnt/HD1/NextPolish/nextPolish", line 369, in main
task.run.start()
File "/home/amit8chiba/miniconda2/lib/python2.7/site-packages/paralleltask/task_control.py", line 347, in start
self._start()
File "/home/amit8chiba/miniconda2/lib/python2.7/site-packages/paralleltask/task_control.py", line 371, in _start
self.submit(job)
File "/home/amit8chiba/miniconda2/lib/python2.7/site-packages/paralleltask/task_control.py", line 255, in submit
_, stdout, _ = self.run(job.cmd)
File "/home/amit8chiba/miniconda2/lib/python2.7/site-packages/paralleltask/task_control.py", line 291, in run
log.critical("Command '%s' returned non-zero exit status %d, error info: %s." % (cmd, p.returncode, stderr))
File "/home/amit8chiba/miniconda2/lib/python2.7/logging/init.py", line 1212, in critical
self._log(CRITICAL, msg, args, **kwargs)
File "/home/amit8chiba/miniconda2/lib/python2.7/logging/init.py", line 1286, in _log
self.handle(record)
File "/home/amit8chiba/miniconda2/lib/python2.7/logging/init.py", line 1296, in handle
self.callHandlers(record)
File "/home/amit8chiba/miniconda2/lib/python2.7/logging/init.py", line 1336, in callHandlers
hdlr.handle(record)
File "/home/amit8chiba/miniconda2/lib/python2.7/logging/init.py", line 759, in handle
self.emit(record)
File "/home/amit8chiba/miniconda2/lib/python2.7/site-packages/paralleltask/kit.py", line 42, in emit
raise Exception(record.msg)
Exception: Command '/bin/sh /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir3/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh > /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir3/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh.o 2>
/mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir3/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh.e' returned non-zero exit status 1, error info: .
[154451 INFO] 2022-03-02 02:47:30 Submitted jobID:[154458] jobCmd:[/mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir3/00.lgs_polish/01.db_split.sh.work/db_split2/nextPolish.sh] in the local_cycle.
[154451 INFO] 2022-03-02 02:47:31 Submitted jobID:[154474] jobCmd:[/mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir3/00.lgs_polish/01.db_split.sh.work/db_split3/nextPolish.sh] in the local_cycle.

What does that mean and how to solve the error. I will be grateful for your advise. Please find below asked information and please let me know if further information is need to help me.

Thank you so much in advance,
Amit

Error message
[153452 INFO] 2022-03-02 02:06:54 NextPolish start...
[153452 INFO] 2022-03-02 02:06:54 version:v1.4.0 logfile:pid153452.log.info
[153452 WARNING] 2022-03-02 02:06:54 Re-write workdir
[153452 WARNING] 2022-03-02 02:06:55 Delete task: 6 due to missing hifi_fofn.
[153452 WARNING] 2022-03-02 02:06:55 Delete task: 6 due to missing hifi_fofn.
[153452 INFO] 2022-03-02 02:06:55 scheduled tasks:
[5, 5, 1, 2, 1, 2]
[153452 INFO] 2022-03-02 02:06:55 options:
[153452 INFO] 2022-03-02 02:06:55
rerun: 3
kill: None
rewrite: 1
cleantmp: 0
use_drmaa: 0
submit: None
job_type: local
sgs_unpaired: 0
sgs_rm_nread: 1
parallel_jobs: 12
align_threads: 48
check_alive: None
job_id_regex: None
lgs_max_depth: 100
lgs_read_type: clr
sgs_max_depth: 100
lgs_max_read_len: 0
hifi_max_depth: 100
multithread_jobs: 10
lgs_min_read_len: 1k
hifi_max_read_len: 0
hifi_block_size: 500M
polish_options: -p 10
hifi_min_read_len: 1k
job_prefix: nextPolish
genome_size: 1802711686
task: [5, 5, 1, 2, 1, 2]
lgs_block_size: 500000000
sgs_block_size: 500000000
sgs_use_duplicate_reads: 0
hifi_minimap2_options: -x map-pb
sgs_align_options: bwa mem -p -t 10
lgs_minimap2_options: -x map-pb -t 48
sgs_fofn: /mnt/md1/Gg_GI_polishing/Sf_polishing/./sgs.fofn
lgs_fofn: /mnt/md1/Gg_GI_polishing/Sf_polishing/./lgs.fofn
workdir: /mnt/md1/Gg_GI_polishing/Sf_polishing/./Sf_Nextpolish_rundir
genome: /mnt/md1/Gg_GI_polishing/Sf_polishing/./Sf_tgs_Gapfilled_assembly.fasta
snp_valid: /mnt/md1/Gg_GI_polishing/Sf_polishing/./Sf_Nextpolish_rundir/%02d.snp_valid
snp_phase: /mnt/md1/Gg_GI_polishing/Sf_polishing/./Sf_Nextpolish_rundir/%02d.snp_phase
lgs_polish: /mnt/md1/Gg_GI_polishing/Sf_polishing/./Sf_Nextpolish_rundir/%02d.lgs_polish
kmer_count: /mnt/md1/Gg_GI_polishing/Sf_polishing/./Sf_Nextpolish_rundir/%02d.kmer_count
hifi_polish: /mnt/md1/Gg_GI_polishing/Sf_polishing/./Sf_Nextpolish_rundir/%02d.hifi_polish
score_chain: /mnt/md1/Gg_GI_polishing/Sf_polishing/./Sf_Nextpolish_rundir/%02d.score_chain
[153452 INFO] 2022-03-02 02:06:55 step 0 and task 5 start:
[153452 INFO] 2022-03-02 02:07:00 Total jobs: 3
[153452 INFO] 2022-03-02 02:07:00 Submitted jobID:[153453] jobCmd:[/mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh] in the local_cycle.
[153453 CRITICAL] 2022-03-02 02:07:00 Command '/bin/sh /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh > /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh.o 2> /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh.e' returned non-zero exit status 1, error info: .
[153452 INFO] 2022-03-02 02:07:01 Submitted jobID:[153459] jobCmd:[/mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir/00.lgs_polish/01.db_split.sh.work/db_split2/nextPolish.sh] in the local_cycle.
[153452 INFO] 2022-03-02 02:07:01 Submitted jobID:[153475] jobCmd:[/mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir/00.lgs_polish/01.db_split.sh.work/db_split3/nextPolish.sh] in the local_cycle.
[153452 ERROR] 2022-03-02 02:45:28 db_split failed: please check the following logs:
[153452 ERROR] 2022-03-02 02:45:28 /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir/00.lgs_polish/01.db_split.sh.work/db_split1/nextPolish.sh.e

nextPolish.sh.e

hostname

• hostname
  cd /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir/00.lgs_polish/01.db_split.sh.work/db_split1
• cd /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir/00.lgs_polish/01.db_split.sh.work/db_split1
  time /mnt/HD1/NextPolish/bin/seq_split -d /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir -m 500000000 -n 12 -t 10 -i 1 -s 180271168600 -p input.sgspart /mnt/md1/Gg_GI_polishing/Sf_polishing/sgs.fofn
• time /mnt/HD1/NextPolish/bin/seq_split -d /mnt/md1/Gg_GI_polishing/Sf_polishing/Sf_Nextpolish_rundir -m 500000000 -n 12 -t 10 -i 1 -s 180271168600 -p input.sgspart /mnt/md1/Gg_GI_polishing/Sf_polishing/sgs.fofn
  Error! /mnt/md1/Gg_GI_polishing/Sf_polishing/Soplha_reverse_paired.fastq.g does not exist!Command exited with non-zero status 1
  0.00user 0.00system 0:00.00elapsed 50%CPU (0avgtext+0avgdata 2136maxresident)k
  0inputs+0outputs (0major+104minor)pagefaults 0swaps

Operating system
Which operating system and version are you using?
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal

GCC
What version of GCC are you using?
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/6/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 6.4.0-17ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr --with-as=/usr/bin/x86_64-linux-gnu-as --with-ld=/usr/bin/x86_64-lin
ux-gnu-ld --program-suffix=-6 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug
--enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --
with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 6.4.0 20180424 (Ubuntu 6.4.0-17ubuntu1)

Python
What version of Python are you using?
Python 2.7.14
(Installed using conda)

NextPolish
What version of NextPolish are you using?
nextPolish v1.4.0

[moold]

As the log says /mnt/md1/Gg_GI_polishing/Sf_polishing/Soplha_reverse_paired.fastq.g does not exist!, it seems you have a typo, the filename ends with gz, not z.

[amit4mchiba]

Thank you so much for the reply.

I checked the files, and it had all the files. So, I still do not know why the error. But one thing that I did differently is that after trimming the Illumina reads, I concatinated forward and reverse reads into as single forward and reverse sequencing dataset, and then used it for running the program. I think that probably this dataset is too big (or bigger than expected by the program), causing into this error.

I re-run the program without concatenating fastq files and running through fofn files, and so far, it is running with out any error. Will keep you posted if any new error emerges.

I have another question. I was just testing differences if I polish my assembly using only short reads or long reads. Ofcourse, long reads run ended quickly, but this also resulted in filling all the gaps (from over 1789 assembly gaps, it ended up 0). Clearly, this is a mistake. I checked your manual and instructions, and it says that it is recommended to split the assemblies at the gaps, then polish it using nextpolish, and then scaffold again. Scaffolding does takes a lot of resources and informations. Is it possible to create a bed file with information related to gaps, then run the next polish after splitting the assemblies, and at the end, place the contigs as it was before using the bed file of original gap? Is it possible to implement here or could you suggest some program to do that.

Thank you so much in advance,

with best regards
Amit

[moold]

Hi, NextPolish required paired-end reads should be in separate files by default, you can switch to single file input with the option -unpaired.
For Is it possible to create a bed file with information related to gaps ... ?, Yes, you can do like that, but probably you have to write a small script for this task.