  Computational methods that predict differential gene expression from histone
modification signals are highly desirable for understanding how histone
modifications control the functional heterogeneity of cells through influencing
differential gene regulation. Recent studies either failed to capture
combinatorial effects on differential prediction or primarily only focused on
cell type-specific analysis. In this paper, we develop a novel attention-based
deep learning architecture, DeepDiff, that provides a unified and end-to-end
solution to model and to interpret how dependencies among histone modifications
control the differential patterns of gene regulation. DeepDiff uses a hierarchy
of multiple Long short-term memory (LSTM) modules to encode the spatial
structure of input signals and to model how various histone modifications
cooperate automatically. We introduce and train two levels of attention jointly
with the target prediction, enabling DeepDiff to attend differentially to
relevant modifications and to locate important genome positions for each
modification. Additionally, DeepDiff introduces a novel deep-learning based
multi-task formulation to use the cell-type-specific gene expression
predictions as auxiliary tasks, encouraging richer feature embeddings in our
primary task of differential expression prediction. Using data from Roadmap
Epigenomics Project (REMC) for ten different pairs of cell types, we show that
DeepDiff significantly outperforms the state-of-the-art baselines for
differential gene expression prediction. The learned attention weights are
validated by observations from previous studies about how epigenetic mechanisms
connect to differential gene expression. Codes and results are available at
\url{deepchrome.org}
