  The conventional encoder-decoder framework for image captioning generally
adopts a single-pass decoding process, which predicts the target descriptive
sentence word by word in temporal order. Despite the great success of this
framework, it still suffers from two serious disadvantages. Firstly, it is
unable to correct the mistakes in the predicted words, which may mislead the
subsequent prediction and result in error accumulation problem. Secondly, such
a framework can only leverage the already generated words but not the possible
future words, and thus lacks the ability of global planning on linguistic
information. To overcome these limitations, we explore a universal two-pass
decoding framework, where a single-pass decoding based model serving as the
Drafting Model first generates a draft caption according to an input image, and
a Deliberation Model then performs the polishing process to refine the draft
caption to a better image description. Furthermore, inspired from the
complementarity between different modalities, we propose a novel Cross
Modification Attention (CMA) module to enhance the semantic expression of the
image features and filter out error information from the draft captions. We
integrate CMA with the decoder of our Deliberation Model and name it as Cross
Modification Attention based Deliberation Model (CMA-DM). We train our proposed
framework by jointly optimizing all trainable components from scratch with a
trade-off coefficient. Experiments on MS COCO dataset demonstrate that our
approach obtains significant improvements over single-pass decoding baselines
and achieves competitive performances compared with other state-of-the-art
two-pass decoding based methods.
