1.20

Download the source code here: samtools-1.20.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

New work and changes:

• Added a --max-depth option to bedcov, for more control over the depth limit used when calculating the pileup. Previously this limit was set at 64000; now it is set to over 2 billion, so effectively all bases will be counted. (PR #1970, fixes #1950. Reported by ellisjj)

• Added mpileup --output-extra RLEN to display the unclipped read length. (PR #1971, feature request #1959. Requested by Feng Tian)

• Improved checking of symbolic flag names (e.g. UNMAP) passed to samtools. (PR #1981, fixes #1977. Reported by Ilya Shlyakhter)

• The samtools consensus --min-depth option now works for the Bayesian mode as well as the simple one. (PR #1989, feature request #1982. Requested by Gautier Richard)

• It's now possible to use the samtools fastq -d tag:val option multiple times, allowing matches on more than one tag/value. It also gets a -D option which allows the values to be listed in a file. (PR #1993, feature request #1958. Requested by Tristan Lefebure)

• Added samtools fixmate -M option to sanity check base modification (ML, MM, MN) tags, and where necessary adjust modification data on hard-clipped records. (PR #1990)

• Made mpileup run faster. (PR #1995)

• samtools import now adds a @PG header to the files it makes. As with other sub-commands, this can be disabled by using --no-PG. (PR #2008. Requested by Steven Leonard)

• The samtools split -d option to split by tag value now works on tags with integer values. (PR #2005, feature request #1956. Requested by Alex Leonard)

• Adjusted samtools sort -n (by name) so that primary reads are always sorted before secondary / supplementary. (PR #2012, feature request #2010. Requested by Stijn van Dongen)

• Added samtools bedcov -H option to print column headers in the output. (PR #2025. Thanks to Dr. K. D. Murray)

Documentation:

• Added a note that BAQ is applied before filtering and overlap removal during mpileup processing. (PR #1988, fixes #1985. Reported by Joseph Galasso)

• Added 3.1 to the list of supported CRAM versions in the samtools manual page. (PR #2009. Thanks to Andrew Thrasher)

• Made assorted improvements to ampliconclip, flagstat and markdup manual pages. (PR #2014)

Bug Fixes:

• Security fix: Fixed double free that could occur if bed file indexing failed due to running out of memory. This bug first appeared in version 1.19.1. (PR #2026)

• Corrected error message printed when faidx fails to load the fai index. (PR #1987. Thanks to Nick Moore)

• Fixed bug introduced in release 1.4 that caused incorrect reference bases to be printed by samtools mpileup -a -f ref.fa in the zero-depth regions at the end of each reference. (PR #2019, fixes #2018. Reported by Joe Georgeson)

• Fixed a samtools view usage crash on MinGW when given invalid options. (PR #2030, fixes #2029. Reported by Divon Lan)

Non user-visible changes and build improvements:

• Added tests to ensure that CRAM compression is working properly. (PR #1969, part of fix for #1968. Reported by Clockris)

1.19.2

Download the source code here: samtools-1.19.2.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

Bug Fixes:

• Fixed a regression in 1.19.1 that broke BED filtering for inputs where the region start positions for the same reference were not sorted in ascending order. (PR #1975, fixes #1974. Reported by Anže Starič)

1.19.1

Download the source code here: samtools-1.19.1.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

Bug Fixes:

• Fixed a possible array bounds violation when looking up regions in a BED file (e.g. using samtools view -L). This could lead to crashes or the return of incomplete results if the BED file contained a large number of entries all referencing low positions on a chromosome. (PR #1962, fixes #1961. Reported by geertvandeweyer)

• Fixed a crash in samtools stats that occurred when trying to clean up after it was unable to open a CRAM reference file. (PR #1957, fixes crash reported in samtools/htslib#1723. Reported by Alex Leonard)
  Documentation:

• Fixed inverted logic in the samtools consensus --show-del manual page description. (PR #1955, fixes #1951. Reported by Mikhail Schelkunov)

• Added a description of the MPC section to the samtools stats manual page. (PR #1963, fixes #1954. Reported by litun-fkby)

1.19

Download the source code here: samtools-1.19.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

New work and changes:

• Samtools coverage: add a new --plot-depth option to draw depth (of coverage) rather than the percentage of bases covered. (PR #1910. Thanks to Pierre Lindenbaum)

• Samtools merge / sort: add a lexicographical name-sort option via the -N option. The "natural" alpha-numeric sort is still available via -n. (PR #1900, fixes #1500. Reported by Steve Huang)

• Samtools view: add -N ^NAME_FILE and -R ^RG_FILE options. The standard -N and -R options only output reads matching a specified list of names or read-groups. With a caret (^) prefix these may be negated to only output read not matching the specified files. (PR #1896, fixes #1895. Suggested by Feng Tian)

• Cope with Htslib's change to no longer close stdout on hts_close(). Htslib companion PR is samtools/htslib#1665. (PR #1909. Thanks to John Marshall)

• Plot-bamstats: add a new plot of the read lengths ("RL") from samtools stats output. (PR #1922, fixes #1824. Thanks to erboone, suggested by Martin Pollard)

• Samtools split: support splitting files based on the contents of auxiliary tags. Also adds a -M option to limit the number of files split can make, to avoid accidental resource over-allocation, and fixes some issues with --write-index. (PR #1222, PR #1933, fixes #1758. Thanks to Valeriu Ohan, suggested by Scott Norton)

Bug Fixes:

• Samtools stats: empty barcode tags are now treated as having no barcode. (PR #1929, fixes #1926. Reported by Jukka Matilainen)

• Samtools cat: add support for non-seekable streams. The file format detection blocked pipes from working before, but now files may be non-seekable such as stdin or a pipe. (PR #1930, fixes #1731. Reported by Julian Hess)

• Samtools mpileup -aa (absolutely all positions) now produces an output even when given an empty input file. (PR #1939. Reported by Chang Y)

• Samtools markdup: speed up optical duplicate tagging on regions with very deep data. (PR #1952)

Documentation:

• Samtools mpileup: add more usage examples to the man page. (PR #1913, fixes #1801)

• Samtools fastq: explicitly document the order that filters apply. (PR #1907)

• Samtools merge: fix example output to use an uppercase @RG PL field. (PR #1917. Thanks to John Marshall. Reported by Michael Macias)

• Add hclen SAM filter documentation. (PR #1902. See also samtools/htslib#1660)

• Samtools consensus: remove the -5 option from documentation. This option was renamed before the consensus subcommand was merged, but accidentally left in the man page. (PR #1924)

samtools release 1.18:

Download the source code here: samtools-1.18.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

New work and changes:

• Add minimiser sort option to collate by an indexed fasta. Expand the minimiser sort to arrange the minimiser values in the same order as they occur in the reference genome. This is acts as an extremely crude and simplistic read aligner that can be used to boost read compression. (PR #1818)

• Add a --duplicate-count option to markdup. Adds the number of duplicates (including itself) to the original read in a dc tag. (PR #1816. Thanks to wulj2)

• Make calmd handle unaligned data or empty files without throwing an error. This is to make pipelines work more smoothly. A warning will still be issued. (PR #1841, fixes #1839. Reported by Filipe G. Vieira)

• Consistent, more comprehensive flag filtering for fasta/fastq. Added --rf/--incl[ude]-flags and long options for -F (--excl[ude]-flags and -f (--require-flags). (PR #1842. Thanks to Devang Thakkar)

• Apply fastq --input-fmt-option settings. Previously any options specified were not being applied to the input file. (PR #1855. Thanks to John Marshall)

• Add fastq -d TAG[:VAL] check. This mirrors view -d and will only output alignments that match TAG (and VAL if specified). (PR #1863, fixes #1854. Requested by Rasmus Kirkegaard)

• Extend import --order TAG to --order TAG:length. If length is specified, the tag format goes from integer to a 0-padded string format. This is a workaround for BAM and CRAM that cannot encode an order tag of over 4 billion records. (PR #1850, fixes #1847. Reported by Feng Tian)

• New -aa mode for consensus. This works like the -aa option in depth and mpileup. The single a reports all bases in contigs covered by alignments. Double aa (or -a -a) reports Ns even for the references with no alignments against them. (PR #1851, fixes #1849. Requested by Tim Fennell)

• Add long option support to samtools index. (PR #1872, fixes #1869. Reported by Jason Bacon)

• Be consistent with rounding of "average length" in samtools stats. (PR #1876, fixes #1867. Reported by Jelinek-J)

• Add option to ampliconclip that marks reads as unmapped when they do not have enough aligned bases left after clipping. Default is to unmap reads with zero aligned bases. (PR #1865, fixes #1856. Requested by ces)

Bug Fixes:

• [From HTSLib] Fix a major bug when searching against a CRAM index where one container has start and end coordinates entirely contained within the previous container. This would occasionally miss data, and sometimes return much more than required. The bug affected versions 1.11 to 1.17, although the change in 1.11 was bug-fixing multi-threaded index queries. This bug did not affect index building. There is no need to reindex your CRAM files. (PR #samtools/htslib#1574, PR #samtools/htslib#1640. Fixes #samtools/htslib#1569, #samtools/htslib#1639, #1808, #1819. Reported by xuxif, Jens Reeder and Jared Simpson)

• Fix a sort -M bug (regression) when merging sub-blocks. Data was valid but in a poor order for compression. (PR #1812)

• Fix bug in split output format. Now SAM and CRAM format can chosen as well as BAM. Also a documentation change, see below. (PR #1821)

• Add error checking to view -e filter expression code. Invalid expressions were not returning an error code. (PR #1833, fixes #1829. Reported by Steve Huang)

• Fix reheader CRAM output version. Sets the correct CRAM output version for non-3.0 CRAMs. (PR #1868, fixes #1866. Reported by John Marshall)

Documentation:

• Expand the default filtering information on the mpileup man page. (PR #1802, fixes #1801. Reported by gevro)

• Add an explanation of the default behaviour of split files on generating a file for reads with missing or unrecognised RG tags. Also a small bug fix, see above. (PR #1821, fixes #1817. Reported by Steve Huang)

• In the INSTALL instructions, switched back to openssl for Alpine. This matches the current Alpine Linux practice. (PR #1837, see htslib#1591. Reported by John Marshall)

• Fix various typos caught by lintian parsers. (PR #1877. Thanks to Étienne Mollier)

• Document consensus --qual-calibration option. (PR #1880, fixes #1879. Reported by John Marshall)

• Updated the page about samtools duplicate marking with more detail at https://www.htslib.org/algorithms/duplicate.html
  Non user-visible changes and build improvements:

• Removed a redundant line that caused a warning in gcc-13. (PR #1838)

1.17

Download the source code here: samtools-1.17.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

New work and changes:

• New samtools reset subcommand. Removes alignment information. Alignment location, CIGAR, mate mapping and flags are updated. If the alignment was in reverse direction, sequence and its quality values are reversed and complemented and the reverse flag is reset. Supplementary and secondary alignment data are discarded. (PR #1767, implements #1682. Requested by dkj)

• New samtools cram-size subcommand. It writes out metrics about a CRAM file reporting aggregate sizes per block "Content ID" fields, the data-series contained within them, and the compression methods used. (PR #1777)

• Added a --sanitize option to fixmate and view. This performs some sanity checks on the state of SAM record fields, fixing up common mistakes made by aligners. (PR #1698)

• Permit 1 thread with samtools view. All other subcommands already allow this and it does provide a modest speed increase. (PR #1755, fixes #1743. Reported by Goran Vinterhalter)

• Add CRAM_OPT_REQUIRED_FIELDS option for view -c. This is a big speed up for CRAM (maybe 5-fold), but it depends on which filtering options are being used. (PR #1776, fixes #1775. Reported by Chang Y)

• New filtering options in samtools depth. The new --excl-flags option is a synonym for -G, with --incl-flags and --require-flags added to match view logic. (PR #1718, fixes #1702. Reported by Dario Beraldi)

• Speed up calmd's slow handling of non-position-sorted data by adding caching. This uses more memory but is only activated when needed. (PR #1723, fixes #1595. Reported by lxwgcool)

• Improve samtools consensus for platforms with instrument specific profiles, considerably helping for data with very different indel error models and providing base quality recalibration tables. On PacBio HiFi, ONT and Ultima Genomics consensus qualities are also redistributed within homopolymers and the likelihood of nearby indel errors is raised. (PR #1721, PR #1733)

• Consensus --mark-ins option. This permits he consensus output to include a markup indicating the next base is an insertion. This is necessary as we need a way of outputting both consensus and also how that consensus marries up with the reference coordinates. (PR #1746)

• Make faidx/fqidx output line length default to the input line length. (PR #1738, fixes #1734. Reported by John Marshall)

• Speed up optical duplicate checking where data has a lot of duplicates compared to non-duplicates. (PR #1779, fixes #1771. Reported by Poshi)

• For collate use TMPDIR environment variable, when looking for a temporary folder. (PR #1782, based on PR #1178 and fixes #1172. Reported by Martin Pollard)

Bug Fixes:

• Fix stats breakage on long deletions when given a reference. (PR #1712, fixes #1707. Reported by John Didion)

• In ampliconclip, stop hard clipping from wrongly removing entire reads. (PR #1722, fixes #1717. Reported by Kevin Xu)

• Fix bug in ampliconstats where references mentioned in the input file headers but not in the bed file would cause it to complain that the SAM headers were inconsistent. (PR #1727, fixes #1650. Reported by jPontix)

• Fixed SEGV in samtools collate when no filename given. (PR #1724)

• Changed the default UMI barcode regex in markdup. The old regex was too restrictive. This version will at least allow the default read name UMI as given in the Illumina example documentation. (PR #1737, fixes #1730. Reported by yloemie)

• Fix samtools consensus buffer overrun with MD:Z handling. (PR #1745, fixes #1744. Reported by trilisser)

• Fix a buffer read-overflow in mpileup and tview on sequences with seq "*". (PR #1747)

• Fix view -X command line parsing that was broken in 1.15. (PR #1772, fixes #1720. Reported by Francisco Rodríguez-Algarra and Miguel Machado)

• Stop samtools view -d from reporting meaningless system errors when tag validation fails. (PR #1796)

Documentation:

• Add a description of the samtools tview display layout to the man page. Documents . vs , and upper vs lowercase. Adds a -s sample example, and documents the -w option. (PR #1765, fixes #1759. Reported by Lucas Ferreira da Silva)

• Clarify intention of samtools fasta/q in man page and soft vs hard clipping. (PR #1794, fixes #1792. Reported by Ryan Lorig-Roach)

• Minor fix to wording of mpileup --rf usage and man page. (PR #1795, fixes #1791. Reported by Luka Pavageau)

Non user-visible changes and build improvements:

• Use POSIX grep in testing as egrep and fgrep are considered obsolete. (PR #1726, thanks to David Seifert)

• Switch MacOS CI tests to an ARM-based image. (PR #1770)

1.16.1

Download the source code here: samtools-1.16.1.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

Bug fixes:

• Fixed a bug with the template-coordinate sort which caused incorrect ordering when using threads, or processing large files that don't fit completely in memory. (PR #1703, thanks to Nils Homer)

• Fixed a crash that occurred when trying to use samtools merge in template-coordinate mode. (PR #1705, thanks to Nils Homer)

1.16

Download the source code here: samtools-1.16.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

New work and changes:

• samtools reference command added. This subcommand extracts the embedded reference out of a CRAM file. (PR #1649, addresses #723. Requested by Torsten Seemann)

• samtools import now adds grouped by query-name to the header. (PR #1633, thanks to Nils Homer)

• Made samtools view read error messages more generic. Former error message would claim that there was a "truncated file or corrupt BAM index file" with no real justification. Also reset errno in stream_view which could lead to confusing error messages. (PR #1645, addresses some of the issues in #1640. Reported by Jian-Guo Zhou)

• Make samtools view -p also clear mqual, tlen and cigar. (PR #1647, fixes #1606. Reported by eboyden)

• Add bedcov option -c to report read count. (PR #1644, fixes #1629. Reported by Natchaphon Rajudom)

• Add UMI/barcode handling to samtools markdup. (PR #1630, fixes #1358 and #1514. Reported by Gert Hulselmans and Poshi)

• Add a new template coordinate sort order to samtools sort and samtools merge. This is useful when working with unique molecular identifiers (UMIs). (PR #1605, fixes #1591. Thanks to Nils Homer)

• Rename mpileup --ignore-overlaps to --ignore-overlaps-removal or --disable-overlap-removal. The previous name was ambiguous and was often read as an option to enable removal of overlapping bases, while in reality this is on by default and the option turns off the ability to remove overlapping bases. (PR #1666, fixes #1663. Reported by yangdingyangding)

• The dict command can now read BWA's .alt file and add AH:* tags indicating reference sequences that represent alternate loci. (PR #1676. Thanks to John Marshall)

• The samtools index command can now accept multiple alignment filenames with the new -M option, and will index each of them separately. (Specifying the output index filename via out.index or the new -o option is currently only applicable when there is only one alignment file to be indexed.) (PR #1674. Reported by Abigail Ramsøe and Nicola Romanò. Thanks to John Marshall)

• Allow samtools fastq -T "*". This allows all tags from SAM records to be written to fastq headers. This is a counterpart to samtools import -T "*". (PR #1679. Thanks to cjw85)

Bug Fixes:

• Re-enable --reference option for samtools depth. The reference is not used but this makes the command line usage compatible with older releases. (PR #1646, fixes #1643. Reported by Randy Harr)

• Fix regex coordinate bug in samtools markdup. (PR #1657, fixes #1642. Reported by Randy Harr)

• Fix divide by zero in plot-bamstats -m, on unmapped data. (PR #1678, fixes #1675. Thanks to Shane McCarthy)

• Fix missing RG headers when using samtools merge -r. (PR #1683, addresses htslib#1479. Reported by Alex Leonard)

• Fix a possible unaligned access in samtools reference. (PR #1696)

Documentation:

• Add documentation on CRAM compression profiles and some of the newer options that appear in CRAM 3.1 and above. (PR #1659, fixes #1656. Reported by Matthias De Smet)

• Add sclen filter expression keyword documentation. (PR #1661, see also htslib#1441)

• Extend FILTER EXPRESSION man page section to match the changes made in HTSlib. (PR #1687, samtools/htslib#1478)

Non user-visible changes and build improvements:

• Ensure generated test files are ignored (by git) and cleaned (by make testclean) (PR #1692, Thanks to John Marshall)

1.15.1

Download the source code here: samtools-1.15.1.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

Bug fixes:

• A bug which prevented the samtools view --region-file (and the equivalent -M -L <file>) options from working in version 1.15 has been fixed. (#1617)

• Fixed a crash triggered by using the samtools view -c/--count and --unmap options together. The --unmap option is now ignored in counting mode. (#1619)

Documentation:

• The consensus command was missing from the main samtools.1 manual page. It has now been added. (#1603)

• Corrected instructions for reproducing the samtools stats "raw total sequences" count using samtools view -c. (#1620; reported by krukanna)

• Improved manual page formatting. (#1625; thanks to John Marshall)

Non user-visible changes and build improvements:

• Unnecessary #include lines have been removed from bam_plcmd.c. (#1607; thanks to John Marshall)

1.15

Download the source code here: samtools-1.15.tar.bz2.(The "Source code" downloads are generated by GitHub and are incomplete as they don't bundle HTSlib and are missing some generated files.)

Notice:

• Samtools mpileup VCF and BCF output (deprecated in release 1.9) has been removed. Please use bcftools mpileup instead.

New work and changes:

• Added --min-BQ and --min-MQ options to depth. These match the equivalent long options found in "samtools mpileup" and gives a consistent way of specifying the base and mapping quality filters. (#1584; fixes #1580. Reported by Chang Y)

• Improved automatic file type detection with view -u or view -1. Setting either of these options would default to BAM format regardless of the usual automatic file type selection based on the file name. The defaults are now only used when the file name does not indicate otherwise. (#1582)

• For markdup optical duplicate marking add regex options for custom coordinates. For the case of non standard read names (QNAME), add options to read the coordinates and, optionally, another part of the string to test for optical duplication. (#1558)

• New samtools consensus subcommand for generating consensus from SAM, BAM or CRAM files based on the contents of the alignment records. The consensus is written as FASTA, FASTQ or as a pileup oriented format. The default FASTA/FASTQ output includes one base per non-gap consensus, with insertions with respect to the aligned reference being included and deletions removed. This could be used to compute a new reference from sequence assemblies to realign against. (#1557)

• New samtools view --fetch-pairs option. This options retrieves pairs even when the mate is outside of the requested region. Using this option enables the multi-region iterator and a region to search must be specified. The input file must be an indexed regular file. (#1542)

• Building on #1530 below, add a tview reflist for Goto. (#1539, thanks to Adam Blanchet)

• Completion of references added to tview Goto. (#1530; thanks to Adam Blanchet)

• New samtools head subcommand for conveniently displaying the headers of a SAM, BAM, or CRAM file. Without options, this is equivalent to samtools view --header-only --no-PG but more succinct and memorable. (#1517; thanks to John Marshall)

Bug Fixes:

• Free memory when stats fails to read the header of a file. (#1592; thanks to Mathias Schmitt)

• Fixed empty field on unsupported aux tags in mpileup --output-extra. Replaces the empty fields on unsupported aux tags with a *. (#1553; fixes #1544. Thanks to Adam Blanchet)

• In mpileup, the --output-BP-5 and --output-BP are no longer mutually exclusive. This fixes the problem of output columns being switched. (#1540; fixes 1534. Reported by Konstantin Riege)

• Fix for hardclip bug in ampliconclip. Odd length sequences resulted in random characters appearing in sequence. (#1538; fixes #1527. Reported by Ivana Mihalek)

Documentation:

• Improved mpileup documentation. (#1566; fixes #1564. Reported by Chang Y)

• Fixed samtools depth -J documentation, which was reversed. (#1552; fixes #1549. Reported by Stephan Hutter)

• Numerous minor man page fixes. (#1528, #1536, #1579, #1590. Thanks to John Marshall for some of these)

Non user-visible changes and build improvements:

• Replace CentOS test build with Rocky Linux. The CentOS Docker images that our test build depended on has stopped working. Switched to Rocky Linux as the nearest available equivalent. (#1589)

• Fix missing autotools on Appveyor. Newer versions of msys2 removed autotools from their base-devel package. This is putting them back. (#1575)

• Fixed bug detected by clang-13 with -Wformat-security. (#1553)

• Switch to using splaysort in bam_lpileup. Improves speed and efficiency in tview. (#1548; thanks to Adam Blanchet)