  Developing and certifying safe - or so-called trustworthy - AI has become an
increasingly salient issue, especially in light of upcoming regulation such as
the EU AI Act. In this context, the black-box nature of machine learning models
limits the use of conventional avenues of approach towards certifying complex
technical systems. As a potential solution, methods to give insights into this
black-box - devised in the field of eXplainable AI (XAI) - could be used. In
this study, the potential and shortcomings of such methods for the purpose of
safe AI development and certification are discussed in 15 qualitative
interviews with experts out of the areas of (X)AI and certification. We find
that XAI methods can be a helpful asset for safe AI development, as they can
show biases and failures of ML-models, but since certification relies on
comprehensive and correct information about technical systems, their impact is
expected to be limited.
