  Large Language Models (LLMs) have the potential to semi-automate some process
mining (PM) analyses. While commercial models are already adequate for many
analytics tasks, the competitive level of open-source LLMs in PM tasks is
unknown. In this paper, we propose PM-LLM-Benchmark, the first comprehensive
benchmark for PM focusing on domain knowledge (process-mining-specific and
process-specific) and on different implementation strategies. We focus also on
the challenges in creating such a benchmark, related to the public availability
of the data and on evaluation biases by the LLMs. Overall, we observe that most
of the considered LLMs can perform some process mining tasks at a satisfactory
level, but tiny models that would run on edge devices are still inadequate. We
also conclude that while the proposed benchmark is useful for identifying LLMs
that are adequate for process mining tasks, further research is needed to
overcome the evaluation biases and perform a more thorough ranking of the
competitive LLMs.
