  For natural language processing systems, two kinds of evidence support the
use of text representations from neural language models "pretrained" on large
unannotated corpora: performance on application-inspired benchmarks (Peters et
al., 2018, inter alia), and the emergence of syntactic abstractions in those
representations (Tenney et al., 2019, inter alia). On the other hand, the lack
of grounded supervision calls into question how well these representations can
ever capture meaning (Bender and Koller, 2020). We apply novel probes to recent
language models -- specifically focusing on predicate-argument structure as
operationalized by semantic dependencies (Ivanova et al., 2012) -- and find
that, unlike syntax, semantics is not brought to the surface by today's
pretrained models. We then use convolutional graph encoders to explicitly
incorporate semantic parses into task-specific finetuning, yielding benefits to
natural language understanding (NLU) tasks in the GLUE benchmark. This approach
demonstrates the potential for general-purpose (rather than task-specific)
linguistic supervision, above and beyond conventional pretraining and
finetuning. Several diagnostics help to localize the benefits of our approach.
