  The ability of semantic reasoning over the sentence pair is essential for
many natural language understanding tasks, e.g., natural language inference and
machine reading comprehension. A recent significant improvement in these tasks
comes from BERT. As reported, the next sentence prediction (NSP) in BERT, which
learns the contextual relationship between two sentences, is of great
significance for downstream problems with sentence-pair input. Despite the
effectiveness of NSP, we suggest that NSP still lacks the essential signal to
distinguish between entailment and shallow correlation. To remedy this, we
propose to augment the NSP task to a 3-class categorization task, which
includes a category for previous sentence prediction (PSP). The involvement of
PSP encourages the model to focus on the informative semantics to determine the
sentence order, thereby improves the ability of semantic understanding. This
simple modification yields remarkable improvement against vanilla BERT. To
further incorporate the document-level information, the scope of NSP and PSP is
expanded into a broader range, i.e., NSP and PSP also include close but
nonsuccessive sentences, the noise of which is mitigated by the label-smoothing
technique. Both qualitative and quantitative experimental results demonstrate
the effectiveness of the proposed method. Our method consistently improves the
performance on the NLI and MRC benchmarks, including the challenging HANS
dataset \cite{hans}, suggesting that the document-level task is still promising
for the pre-training.
