  Predictive business process monitoring (PPM) has been around for several
years as a use case of process mining. PPM enables foreseeing the future of a
business process through predicting relevant information about how a running
process instance might end, related performance indicators, and other
predictable aspects. A big share of PPM approaches adopts a Machine Learning
(ML) technique to address a prediction task, especially non-process-aware PPM
approaches. Consequently, PPM inherits the challenges faced by ML approaches.
One of these challenges concerns the need to gain user trust in the predictions
generated. The field of explainable artificial intelligence (XAI) addresses
this issue. However, the choices made, and the techniques employed in a PPM
task, in addition to ML model characteristics, influence resulting
explanations. A comparison of the influence of different settings on the
generated explanations is missing. To address this gap, we investigate the
effect of different PPM settings on resulting data fed into an ML model and
consequently to a XAI method. We study how differences in resulting
explanations may indicate several issues in underlying data. We construct a
framework for our experiments including different settings at each stage of PPM
with XAI integrated as a fundamental part. Our experiments reveal several
inconsistencies, as well as agreements, between data characteristics (and hence
expectations about these data), important data used by the ML model as a result
of querying it, and explanations of predictions of the investigated ML model.
