  Process mining allows analysts to exploit logs of historical executions of
business processes to extract insights regarding the actual performance of
these processes. One of the most widely studied process mining operations is
automated process discovery. An automated process discovery method takes as
input an event log, and produces as output a business process model that
captures the control-flow relations between tasks that are observed in or
implied by the event log. Various automated process discovery methods have been
proposed in the past two decades, striking different tradeoffs between
scalability, accuracy and complexity of the resulting models. However, these
methods have been evaluated in an ad-hoc manner, employing different datasets,
experimental setups, evaluation measures and baselines, often leading to
incomparable conclusions and sometimes unreproducible results due to the use of
closed datasets. This article provides a systematic review and comparative
evaluation of automated process discovery methods, using an open-source
benchmark and covering twelve publicly-available real-life event logs, twelve
proprietary real-life event logs, and nine quality metrics. The results
highlight gaps and unexplored tradeoffs in the field, including the lack of
scalability of some methods and a strong divergence in their performance with
respect to the different quality metrics used.
