  We do not speak word by word from scratch; our brain quickly structures a
pattern like \textsc{sth do sth at someplace} and then fill in the detailed
descriptions. To render existing encoder-decoder image captioners such
human-like reasoning, we propose a novel framework: learning to Collocate
Neural Modules (CNM), to generate the `inner pattern' connecting visual encoder
and language decoder. Unlike the widely-used neural module networks in visual
Q\&A, where the language (ie, question) is fully observable, CNM for captioning
is more challenging as the language is being generated and thus is partially
observable. To this end, we make the following technical contributions for CNM
training: 1) compact module design --- one for function words and three for
visual content words (eg, noun, adjective, and verb), 2) soft module fusion and
multi-step module execution, robustifying the visual reasoning in partial
observation, 3) a linguistic loss for module controller being faithful to
part-of-speech collocations (eg, adjective is before noun). Extensive
experiments on the challenging MS-COCO image captioning benchmark validate the
effectiveness of our CNM image captioner. In particular, CNM achieves a new
state-of-the-art 127.9 CIDEr-D on Karpathy split and a single-model 126.0 c40
on the official server. CNM is also robust to few training samples, eg, by
training only one sentence per image, CNM can halve the performance loss
compared to a strong baseline.
