  Recent works have recognized the need for human-centered perspectives when
designing and evaluating human-AI interactions and explainable AI methods. Yet,
current approaches fall short at intercepting and managing unexpected user
behavior resulting from the interaction with AI systems and explainability
methods of different stake-holder groups. In this work, we explore the use of
AI and explainability methods in the insurance domain. In an qualitative case
study with participants with different roles and professional backgrounds, we
show that AI and explainability methods are used in creative ways in daily
workflows, resulting in a divergence between their intended and actual use.
Finally, we discuss some recommendations for the design of human-AI
interactions and explainable AI methods to manage the risks and harness the
potential of unexpected user behavior.
