  Knowledge-intensive visual question answering requires models to effectively
use external knowledge to help answer visual questions. A typical pipeline
includes a knowledge retriever and an answer generator. However, a retriever
that utilizes local information, such as an image patch, may not provide
reliable question-candidate relevance scores. Besides, the two-tower
architecture also limits the relevance score modeling of a retriever to select
top candidates for answer generator reasoning. In this paper, we introduce an
additional module, a multi-modal reranker, to improve the ranking quality of
knowledge candidates for answer generation. Our reranking module takes
multi-modal information from both candidates and questions and performs
cross-item interaction for better relevance score modeling. Experiments on
OK-VQA and A-OKVQA show that multi-modal reranker from distant supervision
provides consistent improvements. We also find a training-testing discrepancy
with reranking in answer generation, where performance improves if training
knowledge candidates are similar to or noisier than those used in testing.
