  Temporary syntactic ambiguities arise when the beginning of a sentence is
compatible with multiple syntactic analyses. We inspect to which extent neural
language models (LMs) exhibit uncertainty over such analyses when processing
temporarily ambiguous inputs, and how that uncertainty is modulated by
disambiguating cues. We probe the LM's expectations by generating from it: we
use stochastic decoding to derive a set of sentence completions, and estimate
the probability that the LM assigns to each interpretation based on the
distribution of parses across completions. Unlike scoring-based methods for
targeted syntactic evaluation, this technique makes it possible to explore
completions that are not hypothesized in advance by the researcher. We apply
this method to study the behavior of two LMs (GPT2 and an LSTM) on three types
of temporary ambiguity, using materials from human sentence processing
experiments. We find that LMs can track multiple analyses simultaneously; the
degree of uncertainty varies across constructions and contexts. As a response
to disambiguating cues, the LMs often select the correct interpretation, but
occasional errors point to potential areas of improvement.
