  Knowledge-based visual question answering (VQA) involves questions that
require world knowledge beyond the image to yield the correct answer. Large
language models (LMs) like GPT-3 are particularly helpful for this task because
of their strong knowledge retrieval and reasoning capabilities. To enable LM to
understand images, prior work uses a captioning model to convert images into
text. However, when summarizing an image in a single caption sentence, which
visual entities to describe are often underspecified. Generic image captions
often miss visual details essential for the LM to answer visual questions
correctly. To address this challenge, we propose PromptCap (Prompt-guided image
Captioning), a captioning model designed to serve as a better connector between
images and black-box LMs. Different from generic captions, PromptCap takes a
natural-language prompt to control the visual entities to describe in the
generated caption. The prompt contains a question that the caption should aid
in answering. To avoid extra annotation, PromptCap is trained by examples
synthesized with GPT-3 and existing datasets. We demonstrate PromptCap's
effectiveness on an existing pipeline in which GPT-3 is prompted with image
captions to carry out VQA. PromptCap outperforms generic captions by a large
margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks
(60.4% on OK-VQA and 59.6% on A-OKVQA). Zero-shot results on WebQA show that
PromptCap generalizes well to unseen domains.
