  The problem of estimating subjective visual properties (SVP) of images (e.g.,
Shoes A is more comfortable than B) is gaining rising attention. Due to its
highly subjective nature, different annotators often exhibit different
interpretations of scales when adopting absolute value tests. Therefore, recent
investigations turn to collect pairwise comparisons via crowdsourcing
platforms. However, crowdsourcing data usually contains outliers. For this
purpose, it is desired to develop a robust model for learning SVP from
crowdsourced noisy annotations. In this paper, we construct a deep SVP
prediction model which not only leads to better detection of annotation
outliers but also enables learning with extremely sparse annotations.
Specifically, we construct a comparison multi-graph based on the collected
annotations, where different labeling results correspond to edges with
different directions between two vertexes. Then, we propose a generalized deep
probabilistic framework which consists of an SVP prediction module and an
outlier modeling module that work collaboratively and are optimized jointly.
Extensive experiments on various benchmark datasets demonstrate that our new
approach guarantees promising results.
