  Unpaired cross-lingual image captioning has long suffered from irrelevancy
and disfluency issues, due to the inconsistencies of the semantic scene and
syntax attributes during transfer. In this work, we propose to address the
above problems by incorporating the scene graph (SG) structures and the
syntactic constituency (SC) trees. Our captioner contains the semantic
structure-guided image-to-pivot captioning and the syntactic structure-guided
pivot-to-target translation, two of which are joined via pivot language. We
then take the SG and SC structures as pivoting, performing cross-modal semantic
structure alignment and cross-lingual syntactic structure alignment learning.
We further introduce cross-lingual&cross-modal back-translation training to
fully align the captioning and translation stages. Experiments on
English-Chinese transfers show that our model shows great superiority in
improving captioning relevancy and fluency.
