  In the field of natural language processing (NLP), Large Language Models
(LLMs) have precipitated a paradigm shift, markedly enhancing performance in
natural language generation tasks. Despite these advancements, the
comprehensive evaluation of LLMs remains an inevitable challenge for the
community. Recently, the utilization of Multiple Choice Question Answering
(MCQA) as a benchmark for LLMs has gained considerable traction. This study
first investigates the limitations of MCQA as an evaluation method for LLMs and
then analyzes the fundamental reason for the limitations of MCQA, that while
LLMs may select the correct answers, it is possible that they also recognize
other wrong options as correct. Finally, we propose a dataset augmenting method
for Multiple-Choice Questions (MCQs), MCQA+, that can more accurately reflect
the performance of the model, which underscores the need for more robust
evaluation mechanisms in assessing the performance of LLMs.
