  Multimodal knowledge graphs (MKGs), which intuitively organize information in
various modalities, can benefit multiple practical downstream tasks, such as
recommendation systems, and visual question answering. However, most MKGs are
still far from complete, which motivates the flourishing of MKG reasoning
models. Recently, with the development of general artificial architectures, the
pretrained transformer models have drawn increasing attention, especially for
multimodal scenarios. However, the research of multimodal pretrained
transformer (MPT) for knowledge graph reasoning (KGR) is still at an early
stage. As the biggest difference between MKG and other multimodal data, the
rich structural information underlying the MKG still cannot be fully leveraged
in existing MPT models. Most of them only utilize the graph structure as a
retrieval map for matching images and texts connected with the same entity.
This manner hinders their reasoning performances. To this end, we propose the
graph Structure Guided Multimodal Pretrained Transformer for knowledge graph
reasoning, termed SGMPT. Specifically, the graph structure encoder is adopted
for structural feature encoding. Then, a structure-guided fusion module with
two different strategies, i.e., weighted summation and alignment constraint, is
first designed to inject the structural information into both the textual and
visual features. To the best of our knowledge, SGMPT is the first MPT model for
multimodal KGR, which mines the structural information underlying the knowledge
graph. Extensive experiments on FB15k-237-IMG and WN18-IMG, demonstrate that
our SGMPT outperforms existing state-of-the-art models, and prove the
effectiveness of the designed strategies.
