  To enable Large Language Models (LLMs) to function as conscious agents with
generalizable reasoning capabilities, it is crucial that they possess the
reasoning ability to comprehend situational changes (transitions) in
distribution triggered by environmental factors or actions from other agents.
Despite its fundamental significance, this ability remains underexplored due to
the complexity of modeling infinite possible changes in an event and their
associated distributions, coupled with the lack of benchmark data with
situational transitions. Addressing these gaps, we propose a novel formulation
of reasoning with distributional changes as a three-step discriminative
process, termed as MetAphysical ReaSoning. We then introduce the first-ever
benchmark, MARS, comprising three tasks corresponding to each step. These tasks
systematically assess LLMs' capabilities in reasoning the plausibility of (i)
changes in actions, (ii) states caused by changed actions, and (iii)
situational transitions driven by changes in action. Extensive evaluations with
20 (L)LMs of varying sizes and methods indicate that all three tasks in this
process pose significant challenges, even for state-of-the-art LLMs and LMs
after fine-tuning. Further analyses reveal potential causes for the
underperformance of LLMs and demonstrate that pre-training them on large-scale
conceptualization taxonomies can potentially enhance their metaphysical
reasoning capabilities. Our data and models are publicly accessible at
https://github.com/HKUST-KnowComp/MARS.
