  In this paper we explore the task of modeling semi-structured object
sequences; in particular, we focus our attention on the problem of developing a
structure-aware input representation for such sequences. Examples of such data
include user activity on websites, machine logs, and many others. This type of
data is often represented as a sequence of sets of key-value pairs over time
and can present modeling challenges due to an ever-increasing sequence length.
We propose a two-part approach, which first considers each key independently
and encodes a representation of its values over time; we then self-attend over
these value-aware key representations to accomplish a downstream task. This
allows us to operate on longer object sequences than existing methods. We
introduce a novel shared-attention-head architecture between the two modules
and present an innovative training schedule that interleaves the training of
both modules with shared weights for some attention heads. Our experiments on
multiple prediction tasks using real-world data demonstrate that our approach
outperforms a unified network with hierarchical encoding, as well as other
methods including a record-centric representation and a flattened
representation of the sequence.
