  Automatic attribute discovery methods have gained in popularity to extract
sets of visual attributes from images or videos for various tasks. Despite
their good performance in some classification tasks, it is difficult to
evaluate whether the attributes discovered by these methods are meaningful and
which methods are the most appropriate to discover attributes for visual
descriptions. In its simplest form, such an evaluation can be performed by
manually verifying whether there is any consistent identifiable visual concept
distinguishing between positive and negative exemplars labelled by an
attribute. This manual checking is tedious, expensive and labour intensive. In
addition, comparisons between different methods could also be problematic as it
is not clear how one could quantitatively decide which attribute is more
meaningful than the others. In this paper, we propose a novel attribute
meaningfulness metric to address this challenging problem. With this metric,
automatic quantitative evaluation can be performed on the attribute sets; thus,
reducing the enormous effort to perform manual evaluation. The proposed metric
is applied to some recent automatic attribute discovery and hashing methods on
four attribute-labelled datasets. To further validate the efficacy of the
proposed method, we conducted a user study. In addition, we also compared our
metric with a semi-supervised attribute discover method using the mixture of
probabilistic PCA. In our evaluation, we gleaned several insights that could be
beneficial in developing new automatic attribute discovery methods.
