  Language models (LMs) are capable of conducting in-context learning for
multiple choice reasoning tasks, but the options in these tasks are treated
equally. As humans often first eliminate wrong options before picking the final
correct answer, we argue a similar two-step strategy can make LMs better at
these tasks. To this end, we present the Process of Elimination (POE), a
two-step scoring method. In the first step, POE scores each option, and
eliminates seemingly wrong options. In the second step, POE masks these wrong
options, and makes the final prediction from the remaining options. Zero-shot
experiments on 8 reasoning tasks illustrate the effectiveness of POE, and a
following analysis finds our method to be especially performant on logical
reasoning tasks. We further analyze the effect of masks, and show that POE
applies to few-shot settings and large language models (LLMs) like ChatGPT.
