VCF 4.5 Future of VCF scaling improvements

VCFv4.5.draft.tex

@@ -17,7 +17,7 @@
 \renewcommand{\thefootnote}{\fnsymbol{footnote}}
 
 \begin{document}
-\input{VCFv4.5.ver}
+\input{VCFv4.5.draft.ver}
 \title{\huge \color{red} DRAFT SPEC SUBJECT TO CHANGE \\ The Variant Call Format Specification \\ \vspace{0.5em} \large VCFv4.5 and BCFv2.2}
 \date{\headdate}
 \maketitle
@@ -441,6 +441,7 @@ \subsubsection{Genotype fields}
 First a FORMAT field is given specifying the data types and order (colon-separated FORMAT keys matching the regular expression \texttt{\^{}[A-Za-z\_][0-9A-Za-z\_.]*\$}, duplicate keys are not allowed).
 This is followed by one data block per sample, with the colon-separated data corresponding to the types specified in the format.
 The first key must always be the genotype (GT) if it is present.
+If LGT key is present, it must be after GT (if also present) and before all others.
 There are no required keys.
 Additional Genotype keys can be defined in the meta-information, however, software support for them is not guaranteed.
 
@@ -476,12 +477,17 @@ \subsubsection{Genotype fields}
       ADR		& R		& Integer	& Read depth for each allele on the reverse strand \\
       DP		& 1		& Integer	& Read depth \\
       EC		& A		& Integer	& Expected alternate allele counts \\
+      END		& 1		& Integer	& End position on CHROM (used with multi-sample $<$*$>$ alleles) \\
       FT		& 1		& String	& Filter indicating if this genotype was ``called'' \\
       GL		& G		& Float		& Genotype likelihoods \\
       GP		& G		& Float		& Genotype posterior probabilities \\
       GQ		& 1		& Integer	& Conditional genotype quality \\
       GT		& 1		& String	& Genotype \\
       HQ		& 2		& Integer	& Haplotype quality \\
+      LAA		& .		& Integer	& Strictly increasing indices into REF and ALT, indicating which alternate alleles are relevant (local) for the current sample \\
+      LAD		& .		& Integer	& Read depth for each of the local alternate alleles listed in LAA \\
+      LGT		& .		& String 	& Genotype against the local alleles \\
+      LPL		& .		& Integer	& Phred-scaled genotype likelihoods rounded to the closest integer for genotypes that involve the local alternative alleles listed in LAA \\
       MQ		& 1		& Integer	& RMS mapping quality \\
       PL		& G		& Integer	& Phred-scaled genotype likelihoods rounded to the closest integer \\
       PP		& G		& Integer	& Phred-scaled genotype posterior probabilities rounded to the closest integer \\
@@ -499,6 +505,7 @@ \subsubsection{Genotype fields}
   \item DP (Integer): Read depth at this position for this sample.
   \item EC (Integer): Comma separated list of expected alternate allele counts for each alternate allele in the same order as listed in the ALT field.
   Typically used in association analyses.
+  \item END (Integer): end position of the $<$*$>$ reference block for this sample.
   \item FT (String): Sample genotype filter indicating if this genotype was ``called'' (similar in concept to the FILTER field).
   Again, use PASS to indicate that all filters have been passed, a semicolon-separated list of codes for filters that fail, or `.' to indicate that filters have not been applied.
   These values should be described in the meta-information in the same way as FILTERs.
@@ -578,6 +585,38 @@ \subsubsection{Genotype fields}
     \end{itemize}
 
   \item HQ (Integer): Haplotype qualities, two comma separated phred qualities.
+  \item LAA is a sorted list of $n$ distinct integers, where $0 \le n \le \left|\mathrm{ALT}\right|$, giving the indices of the alleles that are observed in the sample.
+  In callsets with many samples, sites may grow to include numerous alternate alleles at the same POS.
+  Usually, few of these alleles are actually observed in any one sample, but each genotype must supply fields like PL and AD for all of the alleles---a very inefficient representation as PL's size is quadratic in the allele count.
+  Similarly, in rare sites, which can be the bulk of the sites, the vast majority of the samples are reference.
+  To prevent this growth in VCF size, one can choose to specify the genotype, allele depth and the genotype likelihood against a subset of ``Local Alleles''.
+  LAA is the strictly increasing index into REF and ALT, pointing out the alleles that are actually in-play for that sample.
+  0 indicates the REF allele and should always be included with the subsequent values being 1-based indexes into ALT.
+  LAD is the depth of the local alleles,
+  LPL is subset of the PL array that pertains to the alleles that are referred to by LAA,
+  LGT is the genotype but referencing the local alleles rather than the global ones.
+  For example, if REF is G, ALT is A,C,T,\verb!<*>! and a genotype only has information about G, C, and \verb!<*>!, one can have LAA=[0,2,4] and thus LPL will be interpreted as pertaining to the alleles [G, C, \verb!<*>!] and not contain likelihood values for genotypes that involve A or T.
+  In this case LGT=0/1 means that the sample is G/C.
+  GQ is still the genotype quality, even when the genotype is given against the local alleles.
+  Note that reordering might be required and care need to be taken to reorder LAD and LPL appropriately.
+  LAA is required in order to interpret LAD, LPL, and LGT.
+  In the following example, the records with the same POS encode the same information (some columns removed for clarity):
+  \begin{tabular}[l]{llllll}
+  	POS &REF&  ALT&FORMAT&sample\\
+  	1&G&A,C,T,\textless*\textgreater& LAA:LGT:LAD:LPL&  2,4:1/1:20,30,10:90,80,0,100,110,120\\
+  	1&G&A,C,T,\textless*\textgreater& GT:AD:PL&  2/2:20,.,30,.,10:90,.,.,80,.,0,.,.,.,.,100,.,110,.,120\\
+  	2&A&C,G,T,\textless*\textgreater& LAA:LGT:LAD:LPL& 0,3:0/1:15,25:40,0,80\\
+  	2&A&C,G,T,\textless*\textgreater& GT:AD:PL&0/3:15,.,.,25,.:40,.,.,.,.,.,0,.,.,80,.,.,.,.\\
+  	3&C&G,T,\textless*\textgreater& LAA:LGT:LAD:LPL& 0,4:0/0:30,1:0,30,80\\
+  	3&C&G,T,\textless*\textgreater& GT:AD:PL& 0/0:30,.,.1:0,.,.,.,.,.,.,.,.,.,30,.,.,.,80\\
+  	4&G&A,T,\textless*\textgreater& LAA:LGT:LAD:LPL& 0:0/0:30:0\\
+  	4&G&A,T,\textless*\textgreater& GT:AD:PL& 0/0:30,.,..:0,.,.,.,.,.,.,.,.,.,.,.,.,.,.\\
+  \end{tabular}
+  \item LAD: is a list of $n$ integers giving read depths (as per AD) for each of the local alleles as listed in LAA.
+  \item LGT: is the genotype, encoded as allele indexes separated by either of $/$ or $\mid$, as with GT, however, the indexes are into the alleles referenced by LAA.
+  So that in the case that LAA is 0,2,3, LGT=0/2 is equivalent to GT=0/3 and LGT=1/2 is equivalent to GT=2/3 (see example above).
+  \item LPL: is a list of $n \choose \mathrm{Ploidy}$ integers giving phred-scaled genotype likelihoods (rounded to the closest integer; as per PL) for all possible genotypes given the set of alleles defined in the LAA local alleles.
+  The precise ordering is defined in the GL paragraph.
   \item MQ (Integer): RMS mapping quality, similar to the version in the INFO field.
   \item PL (Integer): The phred-scaled genotype likelihoods rounded to the closest integer, and otherwise defined in the same way as the GL field.
   \item PP (Integer): The phred-scaled genotype posterior probabilities rounded to the closest integer, and otherwise defined in the same way as the GP field.
@@ -589,7 +628,7 @@ \subsubsection{Genotype fields}
   All phased genotypes that do not contain a PS subfield are assumed to belong to the same phased set.
   If the genotype in the GT field is unphased, the corresponding PS field is ignored.
   The recommended convention is to use the position of the first variant in the set as the PS identifier (although this is not required).
-  \item PSL (List of Strings): The list of phase sets, one for each allele specified in the {\tt GT}. 
+  \item PSL (List of Strings): The list of phase sets, one for each allele specified in the {\tt GT} or {\tt LGT}. 
   Unphased alleles (without a $\mid$ separator before them) must have the value '$.$' in their corresponding position in the list.
   Unlike {\tt PS} (which is defined per CHROM), records with different CHROM but the same phase-set name are considered part of the same phase set.
   If an implementation cannot guarantee uniqueness of phase-set names across the VCF (for example, phasing a streaming VCF or each CHROM is processed independently in parallel), new phase-set names should be of the format CHROM*POS*ALLELE-NUMBER of the ``first'' allele which is included in this set, with ALLELE-NUMBER being the index of the allele in the {\tt GT} field, since multiple distinct phase-sets could start at the same position. \footnote{The `*' character is used as a separator since `:' is not reserved in the CHROM column.}
@@ -1702,6 +1741,26 @@ \subsection{Representing unspecified alleles and REF-only blocks (gVCF)}
 \normalsize
 
 
+\subsubsection{Multi-sample REF-only blocks}
+When handling VCFs with multiple samples, the length of the $<$*$>$ reference blocks can differ.
+To account for this, a sample-specific END can be specified via the FORMAT END field.
+If any FORMAT END value exists, the INFO END must be present and equal the largest FORMAT END value.
+Positions implicitly called by a preceding $<$*$>$ for a sample must have $GT$/$LGT$ set to the missing value (`.') and have no other FORMAT fields present.
+If $LAA$ is present and a reference block is defined for a given sample, the $<$*$>$ allele must be included as an $LAA$ allele for that sample even though the $LGT$ is $0/0$.
+
+For example, the genotype-only version of the above example with a second sample with no variants:
+\scriptsize
+\begin{flushleft}
+\begin{tabular}{ l l l l l l l l }
+POS & REF & ALT & INFO & FORMAT & SampleA & SampleB \\
+4370 & G & $<$*$>$ & END=4416 & LGT:LAA:END & 0/0:0,1:4388 & 0/0:0,1:4416 \\
+4389 & T & TC & . & LGT:LAA:END & 0/1:0,1:. & . \\
+4390 & C & $<$*$>$ & END=4416 & LGT:LAA:END & 0/0:0,1:4416 & . \\
+\end{tabular}
+\end{flushleft}
+\normalsize
+
+
 \pagebreak
 \subsection{Representing copy number variation}
 \label{cnv}
@@ -2552,6 +2611,8 @@ \section{List of changes}
 \subsection{Changes between VCFv4.5 and VCFv4.4}
 
 \begin{itemize}
+	\item Added local allele support (FORMAT LAA, LGT, LAD, LPL) to reduce the size of multi-sample VCFs and enable lossless merging.
+	\item Added FORMAT END to support sample-specific $<$*$>$ alleles.
 \end{itemize}
 
 \subsection{Changes between VCFv4.4 and VCFv4.3}