  Reasoning about actions and change (RAC) has historically driven the
development of many early AI challenges, such as the frame problem, and many AI
disciplines, including non-monotonic and commonsense reasoning. The role of RAC
remains important even now, particularly for tasks involving dynamic
environments, interactive scenarios, and commonsense reasoning. Despite the
progress of Large Language Models (LLMs) in various AI domains, their
performance on RAC is underexplored. To address this gap, we introduce a new
benchmark, ActionReasoningBench, encompassing 13 domains and rigorously
evaluating LLMs across eight different areas of RAC. These include - Object
Tracking, Fluent Tracking, State Tracking, Action Executability, Effects of
Actions, Numerical RAC, Hallucination Detection, and Composite Questions.
Furthermore, we also investigate the indirect effect of actions due to
ramification constraints for every domain. Finally, we evaluate our benchmark
using open-sourced and commercial state-of-the-art LLMs, including GPT-4o,
Gemini-1.0-Pro, Llama2-7b-chat, Llama2-13b-chat, Llama3-8b-instruct,
Gemma-2b-instruct, and Gemma-7b-instruct. Our findings indicate that these
models face significant challenges across all categories included in our
benchmark.
