  One of the key factors in language productivity and human cognition is the
ability of systematic compositionality, which refers to understanding composed
unseen examples of seen primitives. However, recent evidence reveals that the
Transformers have difficulty generalizing the composed context based on the
seen primitives. To this end, we take the first step to propose a
compositionality-aware Transformer called CAT and two novel pre-training tasks
to facilitate systematic compositionality. We tentatively provide a successful
implementation of a multi-layer CAT on the basis of the especially popular
BERT. The experimental results demonstrate that CAT outperforms baselines on
compositionality-aware tasks with minimal impact on the effectiveness on
standardized language understanding tasks.
