  Reasoning about actions and change (RAC) is essential to understand and
interact with the ever-changing environment. Previous AI research has shown the
importance of fundamental and indispensable knowledge of actions, i.e.,
preconditions and effects. However, traditional methods rely on logical
formalization which hinders practical applications. With recent
transformer-based language models (LMs), reasoning over text is desirable and
seemingly feasible, leading to the question of whether LMs can effectively and
efficiently learn to solve RAC problems. We propose four essential RAC tasks as
a comprehensive textual benchmark and generate problems in a way that minimizes
the influence of other linguistic requirements (e.g., grounding) to focus on
RAC. The resulting benchmark, TRAC, encompassing problems of various
complexities, facilitates a more granular evaluation of LMs, precisely
targeting the structural generalization ability much needed for RAC.
Experiments with three high-performing transformers indicates that additional
efforts are needed to tackle challenges raised by TRAC.
