  We propose a method to create document representations that reflect their
internal structure. We modify Tree-LSTMs to hierarchically merge basic elements
such as words and sentences into blocks of increasing complexity. Our Structure
Tree-LSTM implements a hierarchical attention mechanism over individual
components and combinations thereof. We thus emphasize the usefulness of
Tree-LSTMs for texts larger than a sentence. We show that structure-aware
encoders can be used to improve the performance of document classification. We
demonstrate that our method is resilient to changes to the basic building
blocks, as it performs well with both sentence and word embeddings. The
Structure Tree-LSTM outperforms all the baselines on two datasets by leveraging
structural clues. We show our model's interpretability by visualizing how our
model distributes attention inside a document. On a third dataset from the
medical domain, our model achieves competitive performance with the state of
the art. This result shows the Structure Tree-LSTM can leverage dependency
relations other than text structure, such as a set of reports on the same
patient.
