  Pre-trained Transformer models have achieved successes in a wide range of NLP
tasks, but are inefficient when dealing with long input sequences. Existing
studies try to overcome this challenge via segmenting the long sequence
followed by hierarchical encoding or post-hoc aggregation. We propose a
synchronization mechanism for hierarchical encoding. Our approach first
identifies anchor tokens across segments and groups them by their roles in the
original input sequence. Then inside Transformer layer, anchor embeddings are
synchronized within their group via a self-attention module. Our approach is a
general framework with sufficient flexibility -- when adapted to a new task, it
is easy to be enhanced with the task-specific anchor definitions. Experiments
on two representative tasks with different types of long input texts,
NarrativeQA summary setting and wild multi-hop reasoning from HotpotQA,
demonstrate that our approach is able to improve the global information
exchange among segments while maintaining efficiency.
