  As humans, we can modify our assumptions about a scene by imagining
alternative objects or concepts in our minds. For example, we can easily
anticipate the implications of the sun being overcast by rain clouds (e.g., the
street will get wet) and accordingly prepare for that. In this paper, we
introduce a new task/dataset called Commonsense Reasoning for Counterfactual
Scene Imagination (CoSIm) which is designed to evaluate the ability of AI
systems to reason about scene change imagination. In this task/dataset, models
are given an image and an initial question-response pair about the image. Next,
a counterfactual imagined scene change (in textual form) is applied, and the
model has to predict the new response to the initial question based on this
scene change. We collect 3.5K high-quality and challenging data instances, with
each instance consisting of an image, a commonsense question with a response, a
description of a counterfactual change, a new response to the question, and
three distractor responses. Our dataset contains various complex scene change
types (such as object addition/removal/state change, event description,
environment change, etc.) that require models to imagine many different
scenarios and reason about the changed scenes. We present a baseline model
based on a vision-language Transformer (i.e., LXMERT) and ablation studies.
Through human evaluation, we demonstrate a large human-model performance gap,
suggesting room for promising future work on this challenging counterfactual,
scene imagination task. Our code and dataset are publicly available at:
https://github.com/hyounghk/CoSIm
