  Deep Learning has already been successfully applied to analyze industrial
sensor data in a variety of relevant use cases. However, the opaque nature of
many well-performing methods poses a major obstacle for real-world deployment.
Explainable AI (XAI) and especially feature attribution techniques promise to
enable insights about how such models form their decision. But the plain
application of such methods often fails to provide truly informative and
problem-specific insights to domain experts. In this work, we focus on the
specific task of detecting faults in rolling element bearings from vibration
signals. We propose a novel and domain-specific feature attribution framework
that allows us to evaluate how well the underlying logic of a model corresponds
with expert reasoning. Utilizing the framework we are able to validate the
trustworthiness and to successfully anticipate the generalization ability of
different well-performing deep learning models. Our methodology demonstrates
how signal processing tools can effectively be used to enhance Explainable AI
techniques and acts as a template for similar problems.
