  The currently dominating artificial intelligence and machine learning
technology, neural networks, builds on inductive statistical learning. Neural
networks of today are information processing systems void of understanding and
reasoning capabilities, consequently, they cannot explain promoted decisions in
a humanly valid form. In this work, we revisit and use fundamental philosophy
of science theories as an analytical lens with the goal of revealing, what can
be expected, and more importantly, not expected, from methods that aim to
explain decisions promoted by a neural network. By conducting a case study we
investigate a selection of explainability method's performance over two mundane
domains, animals and headgear. Through our study, we lay bare that the
usefulness of these methods relies on human domain knowledge and our ability to
understand, generalise and reason. The explainability methods can be useful
when the goal is to gain further insights into a trained neural network's
strengths and weaknesses. If our aim instead is to use these explainability
methods to promote actionable decisions or build trust in ML-models they need
to be less ambiguous than they are today. In this work, we conclude from our
study, that benchmarking explainability methods, is a central quest towards
trustworthy artificial intelligence and machine learning.
