  Various galaxy merger detection methods have been applied to diverse
datasets. However, it is difficult to understand how they compare. We aim to
benchmark the relative performance of machine learning (ML) merger detection
methods. We explore six leading ML methods using three main datasets. The first
one (the training data) consists of mock observations from the IllustrisTNG
simulations and allows us to quantify the performance metrics of the detection
methods. The second one consists of mock observations from the Horizon-AGN
simulations, introduced to evaluate the performance of classifiers trained on
different, but comparable data. The third one consists of real observations
from the Hyper Suprime-Cam Subaru Strategic Program (HSC-SSP) survey. For the
binary classification task (mergers vs. non-mergers), all methods perform
reasonably well in the domain of the training data. At $0.1<z<0.3$, precision
and recall range between $\sim$70\% and 80\%, both of which decrease with
increasing $z$ as expected (by $\sim$5\% for precision and $\sim$10\% for
recall at $0.76<z<1.0$). When transferred to a different domain, the precision
of all classifiers is only slightly reduced, but the recall is significantly
worse (by $\sim$20-40\% depending on the method). Zoobot offers the best
overall performance in terms of precision and F1 score. When applied to real
HSC observations, all methods agree well with visual labels of clear mergers
but can differ by more than an order of magnitude in predicting the overall
fraction of major mergers. For the multi-class classification task to
distinguish pre-, post- and non-mergers, none of the methods offer a good
performance, which could be partly due to limitations in resolution and depth
of the data. With the advent of better quality data (e.g. JWST and Euclid), it
is important to improve our ability to detect mergers and distinguish between
merger stages.
