  Recent work on transformer-based neural networks has led to impressive
advances on multiple-choice natural language understanding (NLU) problems, such
as Question Answering (QA) and abductive reasoning. Despite these advances,
there is limited work still on understanding whether these models respond to
perturbed multiple-choice instances in a sufficiently robust manner that would
allow them to be trusted in real-world situations. We present four confusion
probes, inspired by similar phenomena first identified in the behavioral
science community, to test for problems such as prior bias and choice
paralysis. Experimentally, we probe a widely used transformer-based
multiple-choice NLU system using four established benchmark datasets. Here we
show that the model exhibits significant prior bias and to a lesser, but still
highly significant degree, choice paralysis, in addition to other problems. Our
results suggest that stronger testing protocols and additional benchmarks may
be necessary before the language models are used in front-facing systems or
decision making with real world consequences.
