  Answering questions that require reading texts in an image is challenging for
current models. One key difficulty of this task is that rare, polysemous, and
ambiguous words frequently appear in images, e.g., names of places, products,
and sports teams. To overcome this difficulty, only resorting to pre-trained
word embedding models is far from enough. A desired model should utilize the
rich information in multiple modalities of the image to help understand the
meaning of scene texts, e.g., the prominent text on a bottle is most likely to
be the brand. Following this idea, we propose a novel VQA approach, Multi-Modal
Graph Neural Network (MM-GNN). It first represents an image as a graph
consisting of three sub-graphs, depicting visual, semantic, and numeric
modalities respectively. Then, we introduce three aggregators which guide the
message passing from one graph to another to utilize the contexts in various
modalities, so as to refine the features of nodes. The updated nodes have
better features for the downstream question answering module. Experimental
evaluations show that our MM-GNN represents the scene texts better and
obviously facilitates the performances on two VQA tasks that require reading
scene texts.
