  In this paper, we explore effective prompting techniques to enhance zero- and
few-shot Visual Question Answering (VQA) performance in contemporary
Vision-Language Models (VLMs). Central to our investigation is the role of
question templates in guiding VLMs to generate accurate answers. We identify
that specific templates significantly influence VQA outcomes, underscoring the
need for strategic template selection. Another pivotal aspect of our study is
augmenting VLMs with image captions, providing them with additional visual cues
alongside direct image features in VQA tasks. Surprisingly, this augmentation
significantly improves the VLMs' performance in many cases, even though VLMs
"see" the image directly! We explore chain-of-thought (CoT) reasoning and find
that while standard CoT reasoning causes drops in performance, advanced methods
like self-consistency can help recover it. Furthermore, we find that text-only
few-shot examples enhance VLMs' alignment with the task format, particularly
benefiting models prone to verbose zero-shot answers. Lastly, to mitigate the
challenges associated with evaluating free-form open-ended VQA responses using
string-matching based VQA metrics, we introduce a straightforward LLM-guided
pre-processing technique to adapt the model responses to the expected
ground-truth answer distribution. In summary, our research sheds light on the
intricacies of prompting strategies in VLMs for VQA, emphasizing the
synergistic use of captions, templates, and pre-processing to enhance model
efficacy.
