  We propose Localized Narratives, a new form of multimodal image annotations
connecting vision and language. We ask annotators to describe an image with
their voice while simultaneously hovering their mouse over the region they are
describing. Since the voice and the mouse pointer are synchronized, we can
localize every single word in the description. This dense visual grounding
takes the form of a mouse trace segment per word and is unique to our data. We
annotated 849k images with Localized Narratives: the whole COCO, Flickr30k, and
ADE20K datasets, and 671k images of Open Images, all of which we make publicly
available. We provide an extensive analysis of these annotations showing they
are diverse, accurate, and efficient to produce. We also demonstrate their
utility on the application of controlled image captioning.
