  Image captioning is a research hotspot where encoder-decoder models combining
convolutional neural network (CNN) and long short-term memory (LSTM) achieve
promising results. Despite significant progress, these models generate
sentences differently from human cognitive styles. Existing models often
generate a complete sentence from the first word to the end, without
considering the influence of the following words on the whole sentence
generation. In this paper, we explore the utilization of a human-like cognitive
style, i.e., building overall cognition for the image to be described and the
sentence to be constructed, for enhancing computer image understanding. This
paper first proposes a Mutual-aid network structure with Bidirectional LSTMs
(MaBi-LSTMs) for acquiring overall contextual information. In the training
process, the forward and backward LSTMs encode the succeeding and preceding
words into their respective hidden states by simultaneously constructing the
whole sentence in a complementary manner. In the captioning process, the LSTM
implicitly utilizes the subsequent semantic information contained in its hidden
states. In fact, MaBi-LSTMs can generate two sentences in forward and backward
directions. To bridge the gap between cross-domain models and generate a
sentence with higher quality, we further develop a cross-modal attention
mechanism to retouch the two sentences by fusing their salient parts as well as
the salient areas of the image. Experimental results on the Microsoft COCO
dataset show that the proposed model improves the performance of
encoder-decoder models and achieves state-of-the-art results.
