  Comprehending the rich semantics in an image and ordering them in linguistic
order are essential to compose a visually-grounded and linguistically coherent
description for image captioning. Modern techniques commonly capitalize on a
pre-trained object detector/classifier to mine the semantics in an image, while
leaving the inherent linguistic ordering of semantics under-exploited. In this
paper, we propose a new recipe of Transformer-style structure, namely
Comprehending and Ordering Semantics Networks (COS-Net), that novelly unifies
an enriched semantic comprehending and a learnable semantic ordering processes
into a single architecture. Technically, we initially utilize a cross-modal
retrieval model to search the relevant sentences of each image, and all words
in the searched sentences are taken as primary semantic cues. Next, a novel
semantic comprehender is devised to filter out the irrelevant semantic words in
primary semantic cues, and meanwhile infer the missing relevant semantic words
visually grounded in the image. After that, we feed all the screened and
enriched semantic words into a semantic ranker, which learns to allocate all
semantic words in linguistic order as humans. Such sequence of ordered semantic
words are further integrated with visual tokens of images to trigger sentence
generation. Empirical evidences show that COS-Net clearly surpasses the
state-of-the-art approaches on COCO and achieves to-date the best CIDEr score
of 141.1% on Karpathy test split. Source code is available at
\url{https://github.com/YehLi/xmodaler/tree/master/configs/image_caption/cosnet}.
