  For machine reading comprehension, the capacity of effectively modeling the
linguistic knowledge from the detail-riddled and lengthy passages and getting
ride of the noises is essential to improve its performance. Traditional
attentive models attend to all words without explicit constraint, which results
in inaccurate concentration on some dispensable words. In this work, we propose
using syntax to guide the text modeling by incorporating explicit syntactic
constraints into attention mechanism for better linguistically motivated word
representations. In detail, for self-attention network (SAN) sponsored
Transformer-based encoder, we introduce syntactic dependency of interest (SDOI)
design into the SAN to form an SDOI-SAN with syntax-guided self-attention.
Syntax-guided network (SG-Net) is then composed of this extra SDOI-SAN and the
SAN from the original Transformer encoder through a dual contextual
architecture for better linguistics inspired representation. To verify its
effectiveness, the proposed SG-Net is applied to typical pre-trained language
model BERT which is right based on a Transformer encoder. Extensive experiments
on popular benchmarks including SQuAD 2.0 and RACE show that the proposed
SG-Net design helps achieve substantial performance improvement over strong
baselines.
