  Studies of image captioning are shifting towards a trend of a fully
end-to-end paradigm by leveraging powerful visual pre-trained models and
transformer-based generation architecture for more flexible model training and
faster inference speed. State-of-the-art approaches simply extract isolated
concepts or attributes to assist description generation. However, such
approaches do not consider the hierarchical semantic structure in the textual
domain, which leads to an unpredictable mapping between visual representations
and concept words. To this end, we propose a novel Progressive Tree-Structured
prototype Network (dubbed PTSN), which is the first attempt to narrow down the
scope of prediction words with appropriate semantics by modeling the
hierarchical textual semantics. Specifically, we design a novel embedding
method called tree-structured prototype, producing a set of hierarchical
representative embeddings which capture the hierarchical semantic structure in
textual space. To utilize such tree-structured prototypes into visual
cognition, we also propose a progressive aggregation module to exploit semantic
relationships within the image and prototypes. By applying our PTSN to the
end-to-end captioning framework, extensive experiments conducted on MSCOCO
dataset show that our method achieves a new state-of-the-art performance with
144.2% (single model) and 146.5% (ensemble of 4 models) CIDEr scores on
`Karpathy' split and 141.4% (c5) and 143.9% (c40) CIDEr scores on the official
online test server. Trained models and source code have been released at:
https://github.com/NovaMind-Z/PTSN.
