  We examine how users perceive the limitations of an AI system when it
encounters a task that it cannot perform perfectly and whether providing
explanations alongside its answers aids users in constructing an appropriate
mental model of the system's capabilities and limitations. We employ a visual
question answer and explanation task where we control the AI system's
limitations by manipulating the visual inputs: during inference, the system
either processes full-color or grayscale images. Our goal is to determine
whether participants can perceive the limitations of the system. We hypothesize
that explanations will make limited AI capabilities more transparent to users.
However, our results show that explanations do not have this effect. Instead of
allowing users to more accurately assess the limitations of the AI system,
explanations generally increase users' perceptions of the system's competence -
regardless of its actual performance.
