  Inspired by the success of Large Language Models in dealing with new tasks
via In-Context Learning (ICL) in NLP, researchers have also developed Large
Vision-Language Models (LVLMs) with ICL capabilities. However, when
implementing ICL using these LVLMs, researchers usually resort to the simplest
way like random sampling to configure the in-context sequence, thus leading to
sub-optimal results. To enhance the ICL performance, in this study, we use
Visual Question Answering (VQA) as case study to explore diverse in-context
configurations to find the powerful ones. Additionally, through observing the
changes of the LVLM outputs by altering the in-context sequence, we gain
insights into the inner properties of LVLMs, improving our understanding of
them. Specifically, to explore in-context configurations, we design diverse
retrieval methods and employ different strategies to manipulate the retrieved
demonstrations. Through exhaustive experiments on three VQA datasets: VQAv2,
VizWiz, and OK-VQA, we uncover three important inner properties of the applied
LVLM and demonstrate which strategies can consistently improve the ICL VQA
performance. Our code is provided in:
https://github.com/GaryJiajia/OFv2_ICL_VQA.
