  The problem of estimating subjective visual properties from image and video
has attracted increasing interest. A subjective visual property is useful
either on its own (e.g. image and video interestingness) or as an intermediate
representation for visual recognition (e.g. a relative attribute). Due to its
ambiguous nature, annotating the value of a subjective visual property for
learning a prediction model is challenging. To make the annotation more
reliable, recent studies employ crowdsourcing tools to collect pairwise
comparison labels because human annotators are much better at ranking two
images/videos (e.g. which one is more interesting) than giving an absolute
value to each of them separately. However, using crowdsourced data also
introduces outliers. Existing methods rely on majority voting to prune the
annotation outliers/errors. They thus require large amount of pairwise labels
to be collected. More importantly as a local outlier detection method, majority
voting is ineffective in identifying outliers that can cause global ranking
inconsistencies. In this paper, we propose a more principled way to identify
annotation outliers by formulating the subjective visual property prediction
task as a unified robust learning to rank problem, tackling both the outlier
detection and learning to rank jointly. Differing from existing methods, the
proposed method integrates local pairwise comparison labels together to
minimise a cost that corresponds to global inconsistency of ranking order. This
not only leads to better detection of annotation outliers but also enables
learning with extremely sparse annotations. Extensive experiments on various
benchmark datasets demonstrate that our new approach significantly outperforms
state-of-the-arts alternatives.
