pipe error with bwa-mem2.avx512bw (2.2.1)

[pashapu43]

Hi,

I am trying to align paired-end HI-C data with bwa-mem2 but it seems there is an issue with header. Here is the command i used:
bwa-mem2.avx512bw mem -5SP -T0 -t50 -R "@rg\tID:E003868R:379:GW2111194210th:3\tSM:Tim_omnic\tPL:ILLUMINA\tLB:Lib1"
pinhic_tim DTG-OmniC-72_90_R1.fastq.gz DTG-OmniC-72_90_R2.fastq.gz|
pairtools parse --min-mapq 30 --walks-policy 5unique --max-inter-align-gap 30 --nproc-in 20 --nproc-out 20 --chroms-path pinhic.genome |
pairtools sort --tmpdir=./pinhic_tmp --nproc 20|pairtools dedup --nproc-in 20 --nproc-out 20 --mark-dups --output-stats stats.txt|
pairtools split --nproc-in 20 --nproc-out 20 --output-pairs mapped.pairs --output-sam -|samtools view -bS -@16 | samtools sort -@16 -o pinhic_aligned.bam;samtools index pinhic_aligned.bam.

The same command works without any issue with bwa mem (0.7.17-r1188).

Here is the SLRUM output:

Executing in AVX512 mode!!

• SA compression enabled with xfactor: 8

• Ref file: pinhic_tim

• Entering FMI_search

• Index file found. Loading index from pinhic_tim.bwt.2bit.64

• Reference seq len for bi-index = 8647430785

• sentinel-index: 3644441659

• Count:
  0, 1
  1, 2364673030
  2, 4323715393
  3, 6282757756
  4, 8647430785

• Reading other elements of the index from files pinhic_tim

• Index prefix: pinhic_tim
  Traceback (most recent call last):
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/bin/pairtools", line 11, in
  sys.exit(cli())
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1157, in call
  return self.main(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1078, in main
  rv = self.invoke(ctx)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
  return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 783, in invoke
  return __callback(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/init.py", line 183, in wrapper
  return func(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/parse.py", line 165, in parse
  parse_py(
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/parse.py", line 176, in parse_py
  input_sam = AlignmentFilePairtoolized(
  File "pysam/libcalignmentfile.pyx", line 748, in pysam.libcalignmentfile.AlignmentFile.cinit
  File "pysam/libcalignmentfile.pyx", line 953, in pysam.libcalignmentfile.AlignmentFile._open
  ValueError: file does not contain alignment data
  WARNING:pairtools:Headerless input, please, add the header by pairtools header generate or pairtools header transfer
  Traceback (most recent call last):
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/bin/pairtools", line 11, in
  sys.exit(cli())
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1157, in call
  return self.main(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1078, in main
  rv = self.invoke(ctx)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
  return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 783, in invoke
  return __callback(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/init.py", line 183, in wrapper
  return func(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/sort.py", line 70, in sort
  sort_py(pairs_path, output, nproc, tmpdir, memory, compress_program, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/sort.py", line 89, in sort_py
  header = headerops.append_new_pg(header, ID=UTIL_NAME, PN=UTIL_NAME)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/lib/headerops.py", line 346, in append_new_pg
  raise Exception("Input file is not valid .pairs, has no header or is empty.")
  Exception: Input file is not valid .pairs, has no header or is empty.
  WARNING:pairtools:Headerless input, please, add the header by pairtools header generate or pairtools header transfer
  WARNING:pairtools:Pairs file appears not to be sorted, dedup might produce wrong results.
  Traceback (most recent call last):
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/bin/pairtools", line 11, in
  sys.exit(cli())
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1157, in call
  return self.main(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1078, in main
  rv = self.invoke(ctx)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
  return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 783, in invoke
  return __callback(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/init.py", line 183, in wrapper
  return func(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/dedup.py", line 326, in dedup
  dedup_py(
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/dedup.py", line 488, in dedup_py
  header = headerops.append_new_pg(header, ID=UTIL_NAME, PN=UTIL_NAME)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/lib/headerops.py", line 346, in append_new_pg
  raise Exception("Input file is not valid .pairs, has no header or is empty.")
  Exception: Input file is not valid .pairs, has no header or is empty.
  WARNING:pairtools:Headerless input, please, add the header by pairtools header generate or pairtools header transfer
  Traceback (most recent call last):
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/bin/pairtools", line 11, in
  sys.exit(cli())
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1157, in call
  return self.main(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1078, in main
  rv = self.invoke(ctx)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
  return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
  return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/click/core.py", line 783, in invoke
  return __callback(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/init.py", line 183, in wrapper
  return func(*args, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/split.py", line 44, in split
  split_py(pairsam_path, output_pairs, output_sam, **kwargs)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/cli/split.py", line 80, in split_py
  header = headerops.append_new_pg(header, ID=UTIL_NAME, PN=UTIL_NAME)
  File "/mnt/users/akhpasha/.conda/envs/mbase/envs/aligners/lib/python3.10/site-packages/pairtools/lib/headerops.py", line 346, in append_new_pg
  raise Exception("Input file is not valid .pairs, has no header or is empty.")
  Exception: Input file is not valid .pairs, has no header or is empty.
  [main_samview] fail to read the header from "-".
  [W::hts_set_opt] Cannot change block size for this format
  samtools sort: failed to read header from "-"
  [E::hts_open_format] Failed to open file "pinhic_aligned.bam" : No such file or directory
  samtools index: failed to open "pinhic_aligned.bam": No such file or directory

[golobor]

did you check if the bwa mem2 command runs well and generates valid .sam/.bam files? For debugging, I recommend running alignment and parsing as separate commands.