  Recent advances in the field of saliency have concentrated on fixation
prediction, with benchmarks reaching saturation. However, there is an extensive
body of works in psychology and neuroscience that describe aspects of human
visual attention that might not be adequately captured by current approaches.
Here, we investigate singleton detection, which can be thought of as a
canonical example of salience. We introduce two novel datasets, one with
psychophysical patterns and one with natural odd-one-out stimuli. Using these
datasets we demonstrate through extensive experimentation that nearly all
saliency algorithms do not adequately respond to singleton targets in synthetic
and natural images. Furthermore, we investigate the effect of training
state-of-the-art CNN-based saliency models on these types of stimuli and
conclude that the additional training data does not lead to a significant
improvement of their ability to find odd-one-out targets. Datasets are
available at http://data.nvision2.eecs.yorku.ca/P3O3/.
