  Large scale self-supervised pre-training of Transformer language models has
advanced the field of Natural Language Processing and shown promise in
cross-application to the biological `languages' of proteins and DNA. Learning
effective representations of DNA sequences using large genomic sequence
corpuses may accelerate the development of models of gene regulation and
function through transfer learning. However, to accurately model cell
type-specific gene regulation and function, it is necessary to consider not
only the information contained in DNA nucleotide sequences, which is mostly
invariant between cell types, but also how the local chemical and structural
`epigenetic state' of chromosomes varies between cell types. Here, we introduce
a Bidirectional Encoder Representations from Transformers (BERT) model that
learns representations based on both DNA sequence and paired epigenetic state
inputs, which we call Epigenomic BERT (or EBERT). We pre-train EBERT with a
masked language model objective across the entire human genome and across 127
cell types. Training this complex model with a previously prohibitively large
dataset was made possible for the first time by a partnership with Cerebras
Systems, whose CS-1 system powered all pre-training experiments. We show
EBERT's transfer learning potential by demonstrating strong performance on a
cell type-specific transcription factor binding prediction task. Our fine-tuned
model exceeds state of the art performance on 4 of 13 evaluation datasets from
ENCODE-DREAM benchmarks and earns an overall rank of 3rd on the challenge
leaderboard. We explore how the inclusion of epigenetic data and task specific
feature augmentation impact transfer learning performance.
