  This paper introduces MalAlgoQA, a novel dataset designed to evaluate the
counterfactual reasoning capabilities of Large Language Models (LLMs) through a
pedagogical approach. The dataset comprises mathematics and reading
comprehension questions, each accompanied by four answer choices and their
corresponding rationales. We focus on the incorrect answer rationales, termed
"malgorithms", which highlights flawed reasoning steps leading to incorrect
answers and offers valuable insights into erroneous thought processes. We also
propose the Malgorithm Identification task, where LLMs are assessed based on
their ability to identify corresponding malgorithm given an incorrect answer
choice. To evaluate the model performance, we introduce two metrics: Algorithm
Identification Accuracy (AIA) for correct answer rationale identification, and
Malgorithm Identification Accuracy (MIA) for incorrect answer rationale
identification. The task is challenging since state-of-the-art LLMs exhibit
significant drops in MIA as compared to AIA. Moreover, we find that the
chain-of-thought prompting technique not only fails to consistently enhance
MIA, but can also lead to underperformance compared to simple prompting. These
findings hold significant implications for the development of more
cognitively-inspired LLMs to improve their counterfactual reasoning abilities,
particularly through a pedagogical perspective where understanding and
rectifying student misconceptions are crucial.
