  Large biological datasets are being produced at a rapid pace and create
substantial storage challenges, particularly in the domain of high-throughput
sequencing (HTS). Most approaches currently used to store HTS data are either
unable to quickly adapt to the requirements of new sequencing or analysis
methods (because they do not support schema evolution), or fail to provide
state of the art compression of the datasets. We have devised new approaches to
store HTS data that support seamless data schema evolution and compress
datasets substantially better than existing approaches. Building on these new
approaches, we discuss and demonstrate how a multi-tier data organization can
dramatically reduce the storage, computational and network burden of
collecting, analyzing, and archiving large sequencing datasets. For instance,
we show that spliced RNA-Seq alignments can be stored in less than 4% the size
of a BAM file with perfect data fidelity. Compared to the previous compression
state of the art, these methods reduce dataset size more than 20% when storing
gene expression and epigenetic datasets. The approaches have been integrated in
a comprehensive suite of software tools (http://goby.campagnelab.org) that
support common analyses for a range of high-throughput sequencing assays.
