  In this paper, we study the response of large models from the BERT family to
incoherent inputs that should confuse any model that claims to understand
natural language. We define simple heuristics to construct such examples. Our
experiments show that state-of-the-art models consistently fail to recognize
them as ill-formed, and instead produce high confidence predictions on them. As
a consequence of this phenomenon, models trained on sentences with randomly
permuted word order perform close to state-of-the-art models. To alleviate
these issues, we show that if models are explicitly trained to recognize
invalid inputs, they can be robust to such attacks without a drop in
performance.
