  Explainability is one of the key elements for building trust in AI systems.
Among numerous attempts to make AI explainable, quantifying the effect of
explanations remains a challenge in conducting human-AI collaborative tasks.
Aside from the ability to predict the overall behavior of AI, in many
applications, users need to understand an AI agent's competency in different
aspects of the task domain. In this paper, we evaluate the impact of
explanations on the user's mental model of AI agent competency within the task
of visual question answering (VQA). We quantify users' understanding of
competency, based on the correlation between the actual system performance and
user rankings. We introduce an explainable VQA system that uses spatial and
object features and is powered by the BERT language model. Each group of users
sees only one kind of explanation to rank the competencies of the VQA model.
The proposed model is evaluated through between-subject experiments to probe
explanations' impact on the user's perception of competency. The comparison
between two VQA models shows BERT based explanations and the use of object
features improve the user's prediction of the model's competencies.
