Skip to content

Bystro genetic analysis (annotation, filtering, statistics)

License

Notifications You must be signed in to change notification settings

bystrogenomics/bystro-v1

 
 

Repository files navigation

Bystro DOI Codacy Badge

TLDR; 1,000x+ faster than VEP, more complete annotation + online search (https://bystro.io) for datasets of up to 47TB (compressed) online, or petabytes offline.

Bystro Performance

Bystro Publication

For datasets and scripts used, please visit github.com/bystro-paper

If using Bystro, please cite Kotlar et al, Genome Biology, 2018

Web Tutorial

Start here: TUTORIAL.md

For most users, we recommend https://bystro.io .

The web app gives full access to all of Bystro's capabilities, provides a convenient search/filtering interface, supports large data sets (tested up to 890GB uncompressed/129GB compressed), and has excellent performance.

Installing Bystro

The easiest way is to run from Docker: docker pull akotlar/bystro:latest && docker run bystro:latest bystro-annotate.pl

Please read: INSTALL.md for instructions on how to download and use Bystro hg19/hg38/etc databases.

Bystro relies on pluggable (via Bystro's YAML config) pre-processors to normalize variant inputs (dealing with VCF issues such as padding), calculate whether a site is a transition or transversion, calculate sample maf, identify hets/homozygotes/missing samples, calculate heterozygosity, homozygosity, missingness, and more.

  1. VCF format: Bystro-Vcf
  2. SNP format: Bystro-SNP
  3. Create your own to support other formats!

Annotation (Output) Field Descriptions

Please read FIELDS.md

The Bystro configuration file

  • The config file describes the state of both the database and the annotation. It's required for annotating or building

  • It has several keys:

    • tracks: The highest level organization for database values. Tracks have a name property, which must be unique, and a type, which must be one of:

      • sparse: A bed file, or any file that can be mapped to chrom, chromStart, and chromEnd columns.

        • This is used for dbSNP, and Clinvar records, but many files can be fit this format.
        • Mapping fields can be managed by the fieldMap key
      • score: A wigFix file.

        • Used for phastCons, phyloP
      • cadd:

        • A CADD file, or Bystro's custom "bed-like" CADD file, which has 2 header lines, and chrom, chromStart, chromEnd columns, followed by standard CADD fields
        • CADD format: http://cadd.gs.washington.edu
      • gene: A UCSC gene track table (ex: knownGene, refGene, sgdGene) stored as a tab separated output, with column names as columns. Conversion from SQL to the expected tab-delimited format is controlled by bin/bystro-utils.pl, which will automatically fetch the requested sql, and generate the tab-delimited output.

        For instance: For a config file that has the following track

        chromosomes:
          - chr1
        tracks:
          tracks:
          - name: refSeq
            type: gene
            utils:
            - args:
                connection:
                  database: hg19
                sql: SELECT r.*, (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.kgID, '')) SEPARATOR
                  ';') FROM kgXref x WHERE x.refseq=r.name) AS kgID, (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.description,
                  '')) SEPARATOR ';') FROM kgXref x WHERE x.refseq=r.name) AS description,
                  (SELECT GROUP_CONCAT(DISTINCT(NULLIF(e.value, '')) SEPARATOR ';') FROM knownToEnsembl
                  e JOIN kgXref x ON x.kgID = e.name WHERE x.refseq = r.name) AS ensemblID,
                  (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.tRnaName, '')) SEPARATOR ';') FROM
                  kgXref x WHERE x.refseq=r.name) AS tRnaName, (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.spID,
                  '')) SEPARATOR ';') FROM kgXref x WHERE x.refseq=r.name) AS spID, (SELECT
                  GROUP_CONCAT(DISTINCT(NULLIF(x.spDisplayID, '')) SEPARATOR ';') FROM kgXref
                  x WHERE x.refseq=r.name) AS spDisplayID, (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.protAcc,
                  '')) SEPARATOR ';') FROM kgXref x WHERE x.refseq=r.name) AS protAcc, (SELECT
                  GROUP_CONCAT(DISTINCT(NULLIF(x.mRNA, '')) SEPARATOR ';') FROM kgXref x WHERE
                  x.refseq=r.name) AS mRNA, (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.rfamAcc,
                  '')) SEPARATOR ';') FROM kgXref x WHERE x.refseq=r.name) AS rfamAcc FROM
                  refGene r WHERE chrom=%chromosomes%;
        

        Running bin/bystro-utils.pl --config <path/to/this/config> will result in the following config:

        chromosomes:
          - chr1
        tracks:
          tracks:
          - name: refSeq
            type: gene
            local_files:
              - hg19.kgXref.chr1.gz
              name: refSeq
              type: gene
              utils:
              - args:
                  connection:
                    database: hg19
                  sql: SELECT r.*, (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.kgID, '')) SEPARATOR
                    ';') FROM kgXref x WHERE x.refseq=r.name) AS kgID, (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.description,
                    '')) SEPARATOR ';') FROM kgXref x WHERE x.refseq=r.name) AS description,
                    (SELECT GROUP_CONCAT(DISTINCT(NULLIF(e.value, '')) SEPARATOR ';') FROM knownToEnsembl
                    e JOIN kgXref x ON x.kgID = e.name WHERE x.refseq = r.name) AS ensemblID,
                    (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.tRnaName, '')) SEPARATOR ';') FROM
                    kgXref x WHERE x.refseq=r.name) AS tRnaName, (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.spID,
                    '')) SEPARATOR ';') FROM kgXref x WHERE x.refseq=r.name) AS spID, (SELECT
                    GROUP_CONCAT(DISTINCT(NULLIF(x.spDisplayID, '')) SEPARATOR ';') FROM kgXref
                    x WHERE x.refseq=r.name) AS spDisplayID, (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.protAcc,
                    '')) SEPARATOR ';') FROM kgXref x WHERE x.refseq=r.name) AS protAcc, (SELECT
                    GROUP_CONCAT(DISTINCT(NULLIF(x.mRNA, '')) SEPARATOR ';') FROM kgXref x WHERE
                    x.refseq=r.name) AS mRNA, (SELECT GROUP_CONCAT(DISTINCT(NULLIF(x.rfamAcc,
                    '')) SEPARATOR ';') FROM kgXref x WHERE x.refseq=r.name) AS rfamAcc FROM
                    refGene r WHERE chrom=%chromosomes%;
                completed: <date fetched>
                name: fetch
        

        hg19.kgXref.chr1.gz will contain:

        bin	name	chrom	strand	txStart	txEnd	cdsStart	cdsEnd	exonCount	exonStarts	exonEnds	score	name2	cdsStartStat	cdsEndStat	exonFrames	kgID	description	ensemblID	tRnaName	spID	spDisplayID	protAcc	mRNA	rfamAcc
        
        0	NM_001376542	chr1	+	66999275	67216822	67000041	67208778	25	66999275,66999928,67091529,67098752,67105459,67108492,67109226,67126195,67133212,67136677,67137626,67138963,67142686,67145360,67147551,67154830,67155872,67161116,67184976,67194946,67199430,67205017,67206340,67206954,67208755,	66999620,67000051,67091593,67098777,67105516,67108547,67109402,67126207,67133224,67136702,67137678,67139049,67142779,67145435,67148052,67154958,67155999,67161176,67185088,67195102,67199563,67205220,67206405,67207119,67216822,	0	SGIP1	cmpl	cmpl	-1,0,1,2,0,0,1,0,0,0,1,2,1,1,1,1,0,1,1,2,2,0,2,1,1,	NA	NA	NA	NA	NA	NA	NA	NA	NA
      • nearest: A pre-calculated gene track that is intersected with a target gene track.

        Example:

        - name: refSeq.gene
          dist: false
          storeNearest: true
          to: txEnd
          type: nearest
          features:
          - name2
          from: txStart
          local_files:
          - hg19.kgXref.chr*.gz
        

        Options:

        • dist: bool
          • Calculate the distance to the nearest target gene record. If the
      • vcf: A VCF v4.* file

    • chromosomes: The allowable chromosomes.

      • Each row of every track must be identified by these chromosomes (during building)
      • Each row of any input file submitted for annotation must also be "" "" (during annotation)
      • However, Bystro is flexible about the chr prefix

      Ex: For the following config

      chromosomes:
        - chr1
        - chr2
        - chr3

      Only chr1, chr2, and chr3 will be accepted. However, Bystro tries to make your life easy

      1. We currently follow UCSC conventions for chromosomes, meaning they should be prepended by chr
      2. Bystro will automatically append chr to chromosomes read from an input file during annotation.
      3. Bystro allows the transformation of any field during building, configurable in the YAML config file for that assembly, making it easy to prepend chr to the source file chromosome field

      Ex: Clinvar doesn't have a chr prefix, so during building we specify:

      tracks:
        - name: clinvar
          build_field_transformations:
            chrom: chr .
          fieldMap:
            Chromosome: chrom

      Here fieldMap allows us to rename header fields, and build_field_transformations allows us to define a prepend operation (chr . can be interpreted as the perl command "chr" . $chrom)

      So: input files do not need to have their chromosomes prepended by chr. Bystro will normalize the name.

      In this example chromosomes 1 and chr1 will be built/annotated, but 1_rand will not.

Directories and Files

These describe where the Bystro database and any source files are located.

  1. files_dir : The parent folder within which each track's local_files are located
  • Bystro automatically checks for local_files at parent/trackName/file

    Ex: For the config file containing

    files_dir: /path/to/files/
    track:
      - name: refSeq
        local_files:
          - hg19.refGene.chr1.gz
          # and more files

    Bystro will expect files in /path/to/files/refSeq/hg19.refGene.chr1.gz

  1. database_dir : Each database is held within database_dir, in a folder of the name assembly

    Ex: For the config file containing

    assembly: hg19
    database_dir: /path/to/databases/

    Bystro will look for the database /path/to/databases/hg19

About

Bystro genetic analysis (annotation, filtering, statistics)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Perl 58.4%
  • Python 34.8%
  • Rust 3.7%
  • Shell 2.4%
  • Cython 0.4%
  • Dockerfile 0.3%