  In the superpixel literature, the comparison of state-of-the-art methods can
be biased by the non-robustness of some metrics to decomposition aspects, such
as the superpixel scale. Moreover, most recent decomposition methods allow to
set a shape regularity parameter, which can have a substantial impact on the
measured performances. In this paper, we introduce an evaluation framework,
that aims to unify the comparison process of superpixel methods. We investigate
the limitations of existing metrics, and propose to evaluate each of the three
core decomposition aspects: color homogeneity, respect of image objects and
shape regularity. To measure the regularity aspect, we propose a new global
regularity measure (GR), which addresses the non-robustness of state-of-the-art
metrics. We evaluate recent superpixel methods with these criteria, at several
superpixel scales and regularity levels. The proposed framework reduces the
bias in the comparison process of state-of-the-art superpixel methods. Finally,
we demonstrate that the proposed GR measure is correlated with the performances
of various applications.
