<p>The standard formats for storing sequence data are <a href="https://en.wikipedia.org/wiki/FASTA_format" rel="noreferrer">fasta</a> and <a href="https://en.wikipedia.org/wiki/FASTQ_format" rel="noreferrer">fastq</a>.  Fasta is used if you only need the raw sequence data, fastq is used if you want to store the sequence data along with the quality information from base calling.  Each of these can be compressed using gzip or another standard compression algorithm.</p>

<p>Typically we want to keep the quality information along with the raw sequence data, but the quality information accounts for half the storage space required.  Some people have developed <a href="https://web.stanford.edu/~mainakch/papers/QualComp_Final_v2.pdf" rel="noreferrer">algorithms</a> for lossy compression of the quality data that allow us to reduce the storage requirements.</p>

<p>If you are interested in storing variant calling data, the standard format for that is <a href="http://www.internationalgenome.org/wiki/Analysis/vcf4.0/" rel="noreferrer">VCF</a>.  VCF is useful if you want to store quality information of the variant calls, genomic positions, and any annotations you might have about the position.  VCFs can be compressed and indexed using <a href="http://www.htslib.org/doc/tabix.html" rel="noreferrer">bgzip and tabix</a>.  Many tools require variant data to be compressed and indexed using these tools.</p>

CC BY-SA 3.0
2017-05-16T19:23:13.733Z
fastq is generally used for storing read information garnered from sequencing. The information in a fastq is certainly not in chromosome or positional order and can be duplicated with multiple reads representing a position.
morgantaschuk
CC BY-SA 3.0
2017-05-17T03:31:41.963Z
I can confirm with FASTA. Even [Matlab's Bioinformatics Toolbox](https://www.mathworks.com/help/bioinfo/ref/fastaread.html) has a function (`fastaread`) for the import of such data. Plus, all of the genome data (at least that I have used) on NCBI is available in [fasta](https://www.ncbi.nlm.nih.gov/nuccore/357579630?report=fasta).
Ashton Hellwig
CC BY-SA 3.0
2017-05-18T09:01:48.537Z
I read the question as being about storing the reference sequence, not sequencing data.
Ian Sudbery
CC BY-SA 3.0
2017-05-18T10:57:29.243Z
Yes, this answer is about raw sequence data from sequencing projects which is a very inefficient format for storing stable, large sequences. FASTA is indeed very popular, but not FASTQ, not for things like genomes.
terdon
CC BY-SA 3.0
2017-05-16T18:11:54.510Z
2017-05-16T18:11:54.510Z
Greg
<p>The standard and the most common sequence format is FASTA for sure. You can compress it with a compressor. For the ~3GB human genome, gzip reduces the size to ~900MB, depending on the option in use.</p>

<p>Another often used format is UCSC's 2-bit format. This format keeps each A/C/G/T with 2 bits. As I remember, it keeps non-A/C/G/T bases and lowercases in two separate lists. These lists basically tell you that bases between offset x and y are all "N"/lowercase. The 2-bit format loses IUB codes that GRCh37 has. UCSC's hg19 differs from GRCh37 at a few bases.</p>

<p>BWA also produces its own 2-bit format with indexing. You can generate it separately with:</p>

<pre><code>bwa fa2pac -f hg19.fa
</code></pre>

<p>Unlike UCSC, BWA keeps all IUB codes but loses letter cases. BWA does not provide utilities to convert its 2-bit representation to FASTA, either.</p>

<p>The 2-bit format typically reduces the file size down to 1/4 of its original size, unless there are too many scattered ambiguous bases. For human genome, you get a file ~784MB in size. You can compress it further with gzip, but that actually doesn't work well. A gzip'd 2-bit file is only ~5-10% smaller.</p>

<p>If you want to achieve an even smaller file size, you can compress the BWT of 2-bit file. This gives you a ~633MB file:</p>

<pre><code>bwa pac2bwtgen hg19.fa.pac tmp.bwt &amp;&amp; gzip tmp.bwt
</code></pre>

<p>A bit-aware compression algorithm may achieve an even higher compression ratio. However, such BWT-based compression prevents you from extracting subsequences. It is probably of little use in practice.</p>

CC BY-SA 3.0
2017-05-30T15:47:37.947Z
BWA replaces ambiguous bases by random nucleotides. See the original BWA paper:

Section 2.7.1 ... "Non-A/C/G/T
bases on the reference genome are converted to random nucleotides. Doing
so may lead to false hits to regions full of ambiguous bases. Fortunately,
the chance that this may happen is very small given relatively long reads.
We tried 2 million 32 bp reads and did not see any reads mapped to poly-N
regions by chance."
Karel Břinda
CC BY-SA 3.0
2017-05-30T16:21:04.407Z
@Karel All the ambiguous bases are kept in the .amb file. BWA can reconstruct every base, well, at least in principle.
user172818
CC BY-SA 3.0
2017-05-30T16:50:40.380Z
Thank you very much for this information!! I have never used .amb files and they seem to be very useful for me. I wish I were aware of them earlier. Btw. I think we have some code to reconstruct the original sequences from BWA .bwt files. During our work on ProPhyle, we played a little bit with this type of compression. Maybe we will create a separate program for bwt2fa.
Karel Břinda
CC BY-SA 3.0
2017-05-16T19:17:38.697Z
2017-05-16T19:17:38.697Z
user172818
<p>There are several things to consider when asking for "the most efficient" way to store data, it all depends on your use case. Do you just need ACGT, or are there also IUPAC codings for combinations? Do you need additional data (like quality values)? What kind of application are you using the data for (does it need to load all at once or in in chunks? Once or multiple times? Sequential or random access? etc.pp)?</p>

<p>E.g., most efficient for:</p>

<ol>
<li>Lowest footprint on disk, without a lot of hassle: use either FASTA or 2bit, but run through standard compressor (gzip, bzip2, others). The literature you want to consult here is that of standard text compression I think. Also of interest <a href="http://mattmahoney.net/dc/text.html" rel="noreferrer">Large Text Compression Benchmark</a></li>
<li>Keeping the file on disk, but ultra-fast loading small subsets into memory, being able to work in memory with character sized entities: a simple dump of the DNA as characters to disk, maybe combined with an index file to know which chromosome starts where. Then use <a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/mmap.html" rel="noreferrer">mmap</a></li>
<li>Storing quality values: See papers like <a href="http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0059190" rel="noreferrer">Compression of FASTQ and SAM Format Sequencing Data</a> or <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3637481/" rel="noreferrer">Sequence squeeze: an open contest for sequence compression</a></li>
<li>Any combination of the above use cases + a lot more</li>
</ol>

CC BY-SA 3.0
2017-05-16T20:03:21.127Z
I would classify 2bit format as 1b. Lowest footprint on disk, permitting some hassle. In order to use it for anything other than storage, it will need to be converted back into plaintext (fasta), or compressed plaintext (e.g. fasta.gz).
morgantaschuk
CC BY-SA 3.0
2017-05-17T09:51:52.530Z
Your case 2 doesn’t require storing in uncompressed character data format. In fact, A compressed index (for instance) can make access *faster* by avoiding [cache thrashing](https://en.wikipedia.org/wiki/Thrashing_(computer_science)).
Konrad Rudolph
CC BY-SA 3.0
2017-05-16T18:31:11.587Z
2017-05-16T18:31:11.587Z
BaCh
<p>I think the question is a bit ambiguous so please excuse this answer that's a bit redundant from the rest of the ones provided.</p>

<p>As others have mentioned, if you want to store a full genome, <a href="https://en.wikipedia.org/wiki/FASTA_format" rel="noreferrer"><code>FASTA</code></a> and <a href="https://genome.ucsc.edu/FAQ/FAQformat.html#format7" rel="noreferrer"><code>2bit</code></a> formats are appropriate.  For some context, <a href="http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/" rel="noreferrer"><code>hg19</code></a> is about 900Mb compressed for the <code>FASTA</code> file and about 780Mb compressed for the <code>2bit</code> file .  <code>hg19</code> is a reference and is haploid so doesn't represent a "full" human genome that would normally have two alleles for the autosome (non-sex chromosomes).</p>

<p>A common format for representing variant information is Variant Call Format (<a href="https://vcftools.github.io/specs.html" rel="noreferrer"><code>VCF</code></a>). The <code>VCF</code> format represent differences from a reference (<code>hg19</code>, say) that can be used to recover the original full sequence by using the reference and the differences encoded in the <code>VCF</code> file.  I've seen <code>VCF</code> files in the range of 100Mb, but a reference file is still needed to recover the full genome sequence which is the range of 800Mb+, as mentioned above.</p>

<p>If you're considering just one "whole genome" in isolation, then the answer is pretty clear: <code>2bit</code> format is probably approaching the entropy limit of the human genome and you probably won't be able to do much better.
The reason why your question is a bit ambiguous is that as soon as you start encoding more than one genome, a population of genomes, say, then you can start exploiting the redundancy of the genome as shared by the population.</p>

<p>For example, say you want to store two "whole genomes".  You could download the <code>hg19</code> reference and download two <code>VCF</code> files which would give around 1Gb worth of data (around 800Mb for the <code>2bit</code> file and around 200Mb for both of the <code>VCF</code> files).  Now you've been able to represent a "whole genome" in 500Mb instead of the 800Mb.  You can see a similar argument for downloading 3 <code>VCF</code> files and more.</p>

<p>The minimum amount of information needed to represent a population of genomes is, as far as I know, unknown, but I would guess in the 2.5Mb-5Mb range.  For example, see <a href="https://academic.oup.com/bioinformatics/article/25/2/274/218156/Human-genomes-as-email-attachments" rel="noreferrer">"Human genomes as email attachments" by Christley, Lu, Li and Xie</a> which claims a 4Mb encoding of a genome.  </p>

<p>Things get tricky because you have to ask what you're claiming as a "whole genome".  <code>VCF</code> files are notoriously bad because older versions of the specification only store high quality differences from reference, throwing away high quality called sections.  If you want to store low quality information, the encoding is now going to depend on the sequencing technology in weird ways.</p>

<p>Insertions, deletions, mobile insertion elements, copy number variants, other structural variants, etc. all complicate this matter further.  <a href="http://biorxiv.org/content/early/2017/01/18/101378" rel="noreferrer">Genome Graphs</a> are trying to tackle at least some of these problems but the focus is on variant calling rather than efficient individual whole genome representation, though perhaps can be adapted in the future.</p>

CC BY-SA 3.0
2017-06-08T20:50:04.880Z
How can there be 2 orders of magnitude differences between the 2bit compression (hundreds of Mb) and the "e-mail attachment" (a few Mb)? Is the "e-mail attachment" case a really impractical storage format so that no one actually uses it? The abstract doesn't say it, but it seems that what they store is actually variations than the full standalone info.
bli
CC BY-SA 3.0
2017-06-13T00:59:49.423Z
@bli, the order of magnitudes difference comes from exploiting the redundancy in a population. Storing one persons (encoded) DNA requires ~800Mb. Storing thousands of peoples takes (probably) a couple megabytes (each). When dealing with a population of DNA data that you want to encode, there are many ways of doing it.  One way is to store variants from a reference. Another is to store a library of short "reads" and then reference that library. The paper is meant as a proof of concept to answer the underlying question of "what is the theoretical minimum space required to store a whole genome".
abetusk
CC BY-SA 3.0
2017-05-17T03:20:00.923Z
2017-05-19T10:36:42.200Z
2017-05-19T10:36:42.200Z
abetusk
<p>In terms of raw storage capacity 2 bits per nucleotide, and then further compressed with standard compression techniques would be the most efficient. However, you'd still have other storage considerations. Like what to do about non-standard bases: like if you want to indicate a gap or ambiguity. </p>

<p>I'd also query if it is really necessary to store them as binary since it reduces the readability of the data. It is quite convenient having a whole bunch of unix and programming tools that can operate on the string level in text files.</p>

CC BY-SA 3.0
2017-05-16T18:16:27.733Z
In terms of storage space '2 bits per nucleotide' and *compressed* would be a way to save resources.
Severian
CC BY-SA 3.0
2017-05-16T18:08:41.067Z
2017-05-16T18:19:38.200Z
2017-05-16T18:19:38.200Z
niallhaslam
<p>In all seriousness, the most efficient way to store DNA sequence data is...you guessed it...in DNA. <a href="http://dx.doi.org/10.1126/science.1226355" rel="nofollow noreferrer">(Church, Gao, and Kasuri, 2012)</a> and others have used DNA synthesis and sequencing as an information writing/reading mechanism.</p>

<p>Practical? Not yet.</p>

<p>Storage efficiency? Unparalleled!</p>

CC BY-SA 3.0
2017-05-22T18:11:10.693Z
2017-05-22T18:11:10.693Z
Daniel Standage
<p>It is not yet standardized, but <a href="https://www.sevenbridges.com/graph/better-reference/" rel="noreferrer">graph format</a> has the potential for being the most space-efficient method for storing genomes.  The idea is this: rather than store a genome as a linear string of sequenced nucleotides, genomes are stored as overlapping graphs, where sequence variants branch off from the reference genome, and then rejoin when the alignment continues.  Basically, you start with a reference genome, and for every subsequent genome added to the graph, only the differences are stored.  This could allow for an enormous gain in space efficiency. </p>

CC BY-SA 3.0
2017-05-16T18:46:51.883Z
2017-05-16T18:46:51.883Z
woemler
<p><a href="https://github.com/divonlan/genozip" rel="nofollow noreferrer">Genozip</a> is lossless compression software that has been optimized for genomic data. If <a href="https://genozip.readthedocs.io/benchmarks.html" rel="nofollow noreferrer">the benchmarks</a> are to be believed, Illumina FASTQ files compressed with Genozip are nearly 5× smaller than FASTQ files compressed with gzip. Equally impressive gains over gzip are seen for FASTA, VCF, and GFF3 files. See the publication in <em>Bioinformatics</em> for more details:</p>
<p><a href="https://academic.oup.com/bioinformatics/article/37/16/2225/6135077" rel="nofollow noreferrer"><strong>Genozip: a universal extensible genomic data compressor</strong></a></p>

CC BY-SA 4.0
2022-02-23T16:01:18.060Z
2022-02-23T16:01:18.060Z
acvill
<p>Genomes are commonly stored as either fasta files (.fa) or twoBit (.2bit) files. Fasta files store the entire sequence as text and are thus not particularly compressed. </p>

<p>twoBit files store each nucleotide in two bits and contain additional metadata that indicates where there's regions containing <code>N</code> (unknown) bases.</p>

<p>For more information, see the documentation on the twoBit format at the <a href="http://genome.ucsc.edu/FAQ/FAQformat.html#format7" rel="noreferrer">UCSC genome browser</a>.</p>

<p>You can convert between twoBit and fasta format using the <a href="https://genome.ucsc.edu/goldenpath/help/twoBit.html" rel="noreferrer">faToTwoBit and twoBitToFa utilities</a>.</p>

<p>For the human genome, you can download it in either fasta or twoBit format here: <a href="http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/" rel="noreferrer">http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/</a></p>

CC BY-SA 3.0
2017-05-16T18:09:58.197Z
2017-05-16T18:21:25.940Z
2017-05-16T18:21:25.940Z
juniper-
<p>I'd like to learn which format is most commonly used for storing the full human genome sequence (4 letters without a quality score) and why.</p>

<p>I assume that storing it in plain-text format would be very inefficient. I expect a binary format would be more appropriate (e.g. 2 bits per nucleotide).</p>

<p>Which format is most common in terms of space efficiency?</p>

CC BY-SA 3.0
2017-07-27T16:55:02.673Z
@GWW Even if you had 5 letters (i.e. with N) you could get away with 3 bits per nucleotide and still have room for 3 more nucleotide encodings, maybe for like U, mC, hmC.
Jon Deaton
CC BY-SA 3.0
2017-05-16T18:11:45.143Z
see: https://www.biostars.org/p/75178/ "Why Don't We Use Binary Format?"
Pierre
CC BY-SA 3.0
2017-05-16T18:13:11.953Z
Also important question: is the goal to create the smallest footprint on disk for an isolated single genome, or a multiple genomes?
woemler
<human-genome><storage><file-formats>
What's the most efficient file format for the storage of DNA sequences?
# What's the most efficient file format for the storage of DNA sequences?

(Asked by: username_0 on 2017-05-16)

I'd like to learn which format is most commonly used for storing the full human genome sequence (4 letters without a quality score) and why.

I assume that storing it in plain-text format would be very inefficient. I expect a binary format would be more appropriate (e.g. 2 bits per nucleotide).

Which format is most common in terms of space efficiency?

**Tags:** human-genome, storage, file-formats
> **Comments:**
> **username_2**: see: https://www.biostars.org/p/75178/ "Why Don't We Use Binary Format?"  
> **username_3**: Also important question: is the goal to create the smallest footprint on disk for an isolated single genome, or a multiple genomes?  
> **username_1**: @GWW Even if you had 5 letters (i.e. with N) you could get away with 3 bits per nucleotide and still have room for 3 more nucleotide encodings, maybe for like U, mC, hmC.  

---

## 9 answers

### Answer by username_19 on 2017-05-16. Score: 42 | Accepted answer

Genomes are commonly stored as either fasta files (.fa) or twoBit (.2bit) files. Fasta files store the entire sequence as text and are thus not particularly compressed.

twoBit files store each nucleotide in two bits and contain additional metadata that indicates where there's regions containing `N` (unknown) bases.

For more information, see the documentation on the twoBit format at the [UCSC genome browser](http://genome.ucsc.edu/FAQ/FAQformat.html#format7).

You can convert between twoBit and fasta format using the [faToTwoBit and twoBitToFa utilities](https://genome.ucsc.edu/goldenpath/help/twoBit.html).

For the human genome, you can download it in either fasta or twoBit format here: <http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/>


---

### Answer by username_4 on 2017-05-16. Score: 25

The standard formats for storing sequence data are [fasta](https://en.wikipedia.org/wiki/FASTA_format) and [fastq](https://en.wikipedia.org/wiki/FASTQ_format). Fasta is used if you only need the raw sequence data, fastq is used if you want to store the sequence data along with the quality information from base calling. Each of these can be compressed using gzip or another standard compression algorithm.

Typically we want to keep the quality information along with the raw sequence data, but the quality information accounts for half the storage space required. Some people have developed [algorithms](https://web.stanford.edu/~mainakch/papers/QualComp_Final_v2.pdf) for lossy compression of the quality data that allow us to reduce the storage requirements.

If you are interested in storing variant calling data, the standard format for that is [VCF](http://www.internationalgenome.org/wiki/Analysis/vcf4.0/). VCF is useful if you want to store quality information of the variant calls, genomic positions, and any annotations you might have about the position. VCFs can be compressed and indexed using [bgzip and tabix](http://www.htslib.org/doc/tabix.html). Many tools require variant data to be compressed and indexed using these tools.

> **Comments:**
> **username_5**: fastq is generally used for storing read information garnered from sequencing. The information in a fastq is certainly not in chromosome or positional order and can be duplicated with multiple reads representing a position.  
> **username_6**: I can confirm with FASTA. Even [Matlab's Bioinformatics Toolbox](https://www.mathworks.com/help/bioinfo/ref/fastaread.html) has a function (`fastaread`) for the import of such data. Plus, all of the genome data (at least that I have used) on NCBI is available in [fasta](https://www.ncbi.nlm.nih.gov/nuccore/357579630?report=fasta).  
> **username_7**: I read the question as being about storing the reference sequence, not sequencing data.  
> **username_8**: Yes, this answer is about raw sequence data from sequencing projects which is a very inefficient format for storing stable, large sequences. FASTA is indeed very popular, but not FASTQ, not for things like genomes.  

---

### Answer by username_9 on 2017-05-16. Score: 20

The standard and the most common sequence format is FASTA for sure. You can compress it with a compressor. For the ~3GB human genome, gzip reduces the size to ~900MB, depending on the option in use.
Another often used format is UCSC's 2-bit format. This format keeps each A/C/G/T with 2 bits. As I remember, it keeps non-A/C/G/T bases and lowercases in two separate lists. These lists basically tell you that bases between offset x and y are all "N"/lowercase. The 2-bit format loses IUB codes that GRCh37 has. UCSC's hg19 differs from GRCh37 at a few bases.
BWA also produces its own 2-bit format with indexing. You can generate it separately with:
bwa fa2pac -f hg19.fa

Unlike UCSC, BWA keeps all IUB codes but loses letter cases. BWA does not provide utilities to convert its 2-bit representation to FASTA, either.
The 2-bit format typically reduces the file size down to 1/4 of its original size, unless there are too many scattered ambiguous bases. For human genome, you get a file ~784MB in size. You can compress it further with gzip, but that actually doesn't work well. A gzip'd 2-bit file is only ~5-10% smaller.
If you want to achieve an even smaller file size, you can compress the BWT of 2-bit file. This gives you a ~633MB file:
bwa pac2bwtgen hg19.fa.pac tmp.bwt && gzip tmp.bwt

A bit-aware compression algorithm may achieve an even higher compression ratio. However, such BWT-based compression prevents you from extracting subsequences. It is probably of little use in practice.

> **Comments:**
> **username_10**: BWA replaces ambiguous bases by random nucleotides. See the original BWA paper:  Section 2.7.1 ... "Non-A/C/G/T bases on the reference genome are converted to random nucleotides. Doing so may lead to false hits to regions full of ambiguous bases. Fortunately, the chance that this may happen is very small given relatively long reads. We tried 2 million 32 bp reads and did not see any reads mapped to poly-N regions by chance."  
> **username_9**: @Karel All the ambiguous bases are kept in the .amb file. BWA can reconstruct every base, well, at least in principle.  
> **username_10**: Thank you very much for this information!! I have never used .amb files and they seem to be very useful for me. I wish I were aware of them earlier. Btw. I think we have some code to reconstruct the original sequences from BWA .bwt files. During our work on ProPhyle, we played a little bit with this type of compression. Maybe we will create a separate program for bwt2fa.  

---

### Answer by username_11 on 2017-05-16. Score: 13

There are several things to consider when asking for "the most efficient" way to store data, it all depends on your use case. Do you just need ACGT, or are there also IUPAC codings for combinations? Do you need additional data (like quality values)? What kind of application are you using the data for (does it need to load all at once or in in chunks? Once or multiple times? Sequential or random access? etc.pp)?

E.g., most efficient for:

1. Lowest footprint on disk, without a lot of hassle: use either FASTA or 2bit, but run through standard compressor (gzip, bzip2, others). The literature you want to consult here is that of standard text compression I think. Also of interest [Large Text Compression Benchmark](http://mattmahoney.net/dc/text.html)
2. Keeping the file on disk, but ultra-fast loading small subsets into memory, being able to work in memory with character sized entities: a simple dump of the DNA as characters to disk, maybe combined with an index file to know which chromosome starts where. Then use [mmap](http://pubs.opengroup.org/onlinepubs/009695399/functions/mmap.html)
3. Storing quality values: See papers like [Compression of FASTQ and SAM Format Sequencing Data](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0059190) or [Sequence squeeze: an open contest for sequence compression](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3637481/)
4. Any combination of the above use cases + a lot more

> **Comments:**
> **username_5**: I would classify 2bit format as 1b. Lowest footprint on disk, permitting some hassle. In order to use it for anything other than storage, it will need to be converted back into plaintext (fasta), or compressed plaintext (e.g. fasta.gz).  
> **username_12**: Your case 2 doesn’t require storing in uncompressed character data format. In fact, A compressed index (for instance) can make access *faster* by avoiding [cache thrashing](https://en.wikipedia.org/wiki/Thrashing_(computer_science)).  

---

### Answer by username_13 on 2017-05-17. Score: 8

I think the question is a bit ambiguous so please excuse this answer that's a bit redundant from the rest of the ones provided.

As others have mentioned, if you want to store a full genome, [`FASTA`](https://en.wikipedia.org/wiki/FASTA_format) and [`2bit`](https://genome.ucsc.edu/FAQ/FAQformat.html#format7) formats are appropriate. For some context, [`hg19`](http://hgdownload.cse.ucsc.edu/goldenPath/hg19/bigZips/) is about 900Mb compressed for the `FASTA` file and about 780Mb compressed for the `2bit` file . `hg19` is a reference and is haploid so doesn't represent a "full" human genome that would normally have two alleles for the autosome (non-sex chromosomes).

A common format for representing variant information is Variant Call Format ([`VCF`](https://vcftools.github.io/specs.html)). The `VCF` format represent differences from a reference (`hg19`, say) that can be used to recover the original full sequence by using the reference and the differences encoded in the `VCF` file. I've seen `VCF` files in the range of 100Mb, but a reference file is still needed to recover the full genome sequence which is the range of 800Mb+, as mentioned above.

If you're considering just one "whole genome" in isolation, then the answer is pretty clear: `2bit` format is probably approaching the entropy limit of the human genome and you probably won't be able to do much better.
The reason why your question is a bit ambiguous is that as soon as you start encoding more than one genome, a population of genomes, say, then you can start exploiting the redundancy of the genome as shared by the population.

For example, say you want to store two "whole genomes". You could download the `hg19` reference and download two `VCF` files which would give around 1Gb worth of data (around 800Mb for the `2bit` file and around 200Mb for both of the `VCF` files). Now you've been able to represent a "whole genome" in 500Mb instead of the 800Mb. You can see a similar argument for downloading 3 `VCF` files and more.

The minimum amount of information needed to represent a population of genomes is, as far as I know, unknown, but I would guess in the 2.5Mb-5Mb range. For example, see ["Human genomes as email attachments" by Christley, Lu, Li and Xie](https://academic.oup.com/bioinformatics/article/25/2/274/218156/Human-genomes-as-email-attachments) which claims a 4Mb encoding of a genome.

Things get tricky because you have to ask what you're claiming as a "whole genome". `VCF` files are notoriously bad because older versions of the specification only store high quality differences from reference, throwing away high quality called sections. If you want to store low quality information, the encoding is now going to depend on the sequencing technology in weird ways.

Insertions, deletions, mobile insertion elements, copy number variants, other structural variants, etc. all complicate this matter further. [Genome Graphs](http://biorxiv.org/content/early/2017/01/18/101378) are trying to tackle at least some of these problems but the focus is on variant calling rather than efficient individual whole genome representation, though perhaps can be adapted in the future.

> **Comments:**
> **username_14**: How can there be 2 orders of magnitude differences between the 2bit compression (hundreds of Mb) and the "e-mail attachment" (a few Mb)? Is the "e-mail attachment" case a really impractical storage format so that no one actually uses it? The abstract doesn't say it, but it seems that what they store is actually variations than the full standalone info.  
> **username_13**: @bli, the order of magnitudes difference comes from exploiting the redundancy in a population. Storing one persons (encoded) DNA requires ~800Mb. Storing thousands of peoples takes (probably) a couple megabytes (each). When dealing with a population of DNA data that you want to encode, there are many ways of doing it.  One way is to store variants from a reference. Another is to store a library of short "reads" and then reference that library. The paper is meant as a proof of concept to answer the underlying question of "what is the theoretical minimum space required to store a whole genome".  

---

### Answer by username_3 on 2017-05-16. Score: 7

It is not yet standardized, but [graph format](https://www.sevenbridges.com/graph/better-reference/) has the potential for being the most space-efficient method for storing genomes. The idea is this: rather than store a genome as a linear string of sequenced nucleotides, genomes are stored as overlapping graphs, where sequence variants branch off from the reference genome, and then rejoin when the alignment continues. Basically, you start with a reference genome, and for every subsequent genome added to the graph, only the differences are stored. This could allow for an enormous gain in space efficiency.


---

### Answer by username_15 on 2017-05-16. Score: 4

In terms of raw storage capacity 2 bits per nucleotide, and then further compressed with standard compression techniques would be the most efficient. However, you'd still have other storage considerations. Like what to do about non-standard bases: like if you want to indicate a gap or ambiguity.

I'd also query if it is really necessary to store them as binary since it reduces the readability of the data. It is quite convenient having a whole bunch of unix and programming tools that can operate on the string level in text files.

> **Comments:**
> **username_16**: In terms of storage space '2 bits per nucleotide' and *compressed* would be a way to save resources.  

---

### Answer by username_17 on 2017-05-22. Score: 3

In all seriousness, the most efficient way to store DNA sequence data is...you guessed it...in DNA. [(Church, Gao, and Kasuri, 2012)](http://dx.doi.org/10.1126/science.1226355) and others have used DNA synthesis and sequencing as an information writing/reading mechanism.

Practical? Not yet.

Storage efficiency? Unparalleled!


---

### Answer by username_18 on 2022-02-23. Score: 2

[Genozip](https://github.com/divonlan/genozip) is lossless compression software that has been optimized for genomic data. If [the benchmarks](https://genozip.readthedocs.io/benchmarks.html) are to be believed, Illumina FASTQ files compressed with Genozip are nearly 5× smaller than FASTQ files compressed with gzip. Equally impressive gains over gzip are seen for FASTA, VCF, and GFF3 files. See the publication in *Bioinformatics* for more details:

[**Genozip: a universal extensible genomic data compressor**](https://academic.oup.com/bioinformatics/article/37/16/2225/6135077)


---

