  To bridge the gap between humans and machines in image understanding and
describing, we need further insight into how people describe a perceived scene.
In this paper, we study the agreement between bottom-up saliency-based visual
attention and object referrals in scene description constructs. We investigate
the properties of human-written descriptions and machine-generated ones. We
then propose a saliency-boosted image captioning model in order to investigate
benefits from low-level cues in language models. We learn that (1) humans
mention more salient objects earlier than less salient ones in their
descriptions, (2) the better a captioning model performs, the better attention
agreement it has with human descriptions, (3) the proposed saliency-boosted
model, compared to its baseline form, does not improve significantly on the MS
COCO database, indicating explicit bottom-up boosting does not help when the
task is well learnt and tuned on a data, (4) a better generalization is,
however, observed for the saliency-boosted model on unseen data.
