  As Archimedes famously said, ``Give me a lever long enough and a fulcrum on
which to place it, and I shall move the world'', in this study, we propose to
use a tiny Language Model (LM), \eg, a Transformer with 67M parameters, to
lever much larger Vision-Language Models (LVLMs) with 9B parameters.
Specifically, we use this tiny \textbf{Lever-LM} to configure effective
in-context demonstration (ICD) sequences to improve the In-Context Learinng
(ICL) performance of LVLMs. Previous studies show that diverse ICD
configurations like the selection and ordering of the demonstrations heavily
affect the ICL performance, highlighting the significance of configuring
effective ICD sequences. Motivated by this and by re-considering the the
process of configuring ICD sequence, we find this is a mirror process of human
sentence composition and further assume that effective ICD configurations may
contain internal statistical patterns that can be captured by Lever-LM. Then a
dataset with effective ICD sequences is constructed to train Lever-LM. After
training, given novel queries, new ICD sequences are configured by the trained
Lever-LM to solve vision-language tasks through ICL. Experiments show that
these ICD sequences can improve the ICL performance of two LVLMs compared with
some strong baselines in Visual Question Answering and Image Captioning,
validating that Lever-LM can really capture the statistical patterns for
levering LVLMs.
