  We propose to Transform Scene Graphs (TSG) into more descriptive captions. In
TSG, we apply multi-head attention (MHA) to design the Graph Neural Network
(GNN) for embedding scene graphs. After embedding, different graph embeddings
contain diverse specific knowledge for generating the words with different
part-of-speech, e.g., object/attribute embedding is good for generating
nouns/adjectives. Motivated by this, we design a Mixture-of-Expert (MOE)-based
decoder, where each expert is built on MHA, for discriminating the graph
embeddings to generate different kinds of words. Since both the encoder and
decoder are built based on the MHA, as a result, we construct a homogeneous
encoder-decoder unlike the previous heterogeneous ones which usually apply
Fully-Connected-based GNN and LSTM-based decoder. The homogeneous architecture
enables us to unify the training configuration of the whole model instead of
specifying different training strategies for diverse sub-networks as in the
heterogeneous pipeline, which releases the training difficulty. Extensive
experiments on the MS-COCO captioning benchmark validate the effectiveness of
our TSG. The code is in: https://github.com/GaryJiajia/TSG.
