  Though linguistic knowledge emerges during large-scale language model
pretraining, recent work attempt to explicitly incorporate human-defined
linguistic priors into task-specific fine-tuning. Infusing language models with
syntactic or semantic knowledge from parsers has shown improvements on many
language understanding tasks. To further investigate the effectiveness of
structural linguistic priors, we conduct empirical study of replacing parsed
graphs or trees with trivial ones (rarely carrying linguistic knowledge e.g.,
balanced tree) for tasks in the GLUE benchmark. Encoding with trivial graphs
achieves competitive or even better performance in fully-supervised and
few-shot settings. It reveals that the gains might not be significantly
attributed to explicit linguistic priors but rather to more feature
interactions brought by fusion layers. Hence we call for attention to using
trivial graphs as necessary baselines to design advanced knowledge fusion
methods in the future.
