  Object removal refers to the process of erasing designated objects from an
image while preserving the overall appearance, and it is one area where image
inpainting is widely used in real-world applications. The performance of an
object remover is quantitatively evaluated by measuring the quality of object
removal results, similar to how the performance of an image inpainter is
gauged. Current works reporting quantitative performance evaluations utilize
original images as references. In this letter, to validate the current
evaluation methods cannot properly evaluate the performance of an object
remover, we create a dataset with object removal ground truth and compare the
evaluations made by the current methods using original images to those
utilizing object removal ground truth images. The disparities between two
evaluation sets validate that the current methods are not suitable for
measuring the performance of an object remover. Additionally, we propose new
evaluation methods tailored to gauge the performance of an object remover. The
proposed methods evaluate the performance through class-wise object removal
results and utilize images without the target class objects as a comparison
set. We confirm that the proposed methods can make judgments consistent with
human evaluators in the COCO dataset, and that they can produce measurements
aligning with those using object removal ground truth in the self-acquired
dataset.
