  Recently it has been shown that without any access to the contextual passage,
multiple choice reading comprehension (MCRC) systems are able to answer
questions significantly better than random on average. These systems use their
accumulated "world knowledge" to directly answer questions, rather than using
information from the passage. This paper examines the possibility of exploiting
this observation as a tool for test designers to ensure that the use of "world
knowledge" is acceptable for a particular set of questions. We propose
information-theory based metrics that enable the level of "world knowledge"
exploited by systems to be assessed. Two metrics are described: the expected
number of options, which measures whether a passage-free system can identify
the answer a question using world knowledge; and the contextual mutual
information, which measures the importance of context for a given question. We
demonstrate that questions with low expected number of options, and hence
answerable by the shortcut system, are often similarly answerable by humans
without context. This highlights that the general knowledge 'shortcuts' could
be equally used by exam candidates, and that our proposed metrics may be
helpful for future test designers to monitor the quality of questions.
