  Modern predictive analytics underpinned by machine learning techniques has
become a key enabler to the automation of data-driven decision making. In the
context of business process management, predictive analytics has been applied
to making predictions about the future state of an ongoing business process
instance, for example, when will the process instance complete and what will be
the outcome upon completion. Machine learning models can be trained on event
log data recording historical process execution to build the underlying
predictive models. Multiple techniques have been proposed so far which encode
the information available in an event log and construct input features required
to train a predictive model. While accuracy has been a dominant criterion in
the choice of various techniques, they are often applied as a black-box in
building predictive models. In this paper, we derive explanations using
interpretable machine learning techniques to compare and contrast the
suitability of multiple predictive models of high accuracy. The explanations
allow us to gain an understanding of the underlying reasons for a prediction
and highlight scenarios where accuracy alone may not be sufficient in assessing
the suitability of techniques used to encode event log data to features used by
a predictive model. Findings from this study motivate the need and importance
to incorporate interpretability in predictive process analytics.
