  The explanation dimension of Artificial Intelligence (AI) based system has
been a hot topic for the past years. Different communities have raised concerns
about the increasing presence of AI in people's everyday tasks and how it can
affect people's lives. There is a lot of research addressing the
interpretability and transparency concepts of explainable AI (XAI), which are
usually related to algorithms and Machine Learning (ML) models. But in
decision-making scenarios, people need more awareness of how AI works and its
outcomes to build a relationship with that system. Decision-makers usually need
to justify their decision to others in different domains. If that decision is
somehow based on or influenced by an AI-system outcome, the explanation about
how the AI reached that result is key to building trust between AI and humans
in decision-making scenarios. In this position paper, we discuss the role of
XAI in decision-making scenarios, our vision of Decision-Making with AI-system
in the loop, and explore one case from the literature about how XAI can impact
people justifying their decisions, considering the importance of building the
human-AI relationship for those scenarios.
