  Transformers have shown great success in learning representations for
language modelling. However, an open challenge still remains on how to
systematically aggregate semantic information (word embedding) with positional
(or temporal) information (word orders). In this work, we propose a new
architecture to aggregate the two sources of information using cascaded
semantic and positional self-attention network (CSPAN) in the context of
document classification. The CSPAN uses a semantic self-attention layer
cascaded with Bi-LSTM to process the semantic and positional information in a
sequential manner, and then adaptively combine them together through a residue
connection. Compared with commonly used positional encoding schemes, CSPAN can
exploit the interaction between semantics and word positions in a more
interpretable and adaptive manner, and the classification performance can be
notably improved while simultaneously preserving a compact model size and high
convergence rate. We evaluate the CSPAN model on several benchmark data sets
for document classification with careful ablation studies, and demonstrate the
encouraging results compared with state of the art.
