  Entity resolution (ER) is the process of identifying records that refer to
the same entities within one or across multiple databases. Numerous techniques
have been developed to tackle ER challenges over the years, with recent
emphasis placed on machine and deep learning methods for the matching phase.
However, the quality of the benchmark datasets typically used in the
experimental evaluations of learning-based matching algorithms has not been
examined in the literature. To cover this gap, we propose four different
approaches to assessing the difficulty and appropriateness of 13 established
datasets: two theoretical approaches, which involve new measures of linearity
and existing measures of complexity, and two practical approaches: the
difference between the best non-linear and linear matchers, as well as the
difference between the best learning-based matcher and the perfect oracle. Our
analysis demonstrates that most of the popular datasets pose rather easy
classification tasks. As a result, they are not suitable for properly
evaluating learning-based matching algorithms. To address this issue, we
propose a new methodology for yielding benchmark datasets. We put it into
practice by creating four new matching tasks, and we verify that these new
benchmarks are more challenging and therefore more suitable for further
advancements in the field.
