  The current research direction in generative models, such as the recently
developed GPT4, aims to find relevant knowledge information for multimodal and
multilingual inputs to provide answers. Under these research circumstances, the
demand for multilingual evaluation of visual question answering (VQA) tasks, a
representative task of multimodal systems, has increased. Accordingly, we
propose a bilingual outside-knowledge VQA (BOK-VQA) dataset in this study that
can be extended to multilingualism. The proposed data include 17K images, 17K
question-answer pairs for both Korean and English and 280K instances of
knowledge information related to question-answer content. We also present a
framework that can effectively inject knowledge information into a VQA system
by pretraining the knowledge information of BOK-VQA data in the form of graph
embeddings. Finally, through in-depth analysis, we demonstrated the actual
effect of the knowledge information contained in the constructed training data
on VQA.
