  This paper presents VisLingInstruct, a novel approach to advancing
Multi-Modal Language Models (MMLMs) in zero-shot learning. Current MMLMs show
impressive zero-shot abilities in multi-modal tasks, but their performance
depends heavily on the quality of instructions. VisLingInstruct tackles this by
autonomously evaluating and optimizing instructional texts through In-Context
Learning, improving the synergy between visual perception and linguistic
expression in MMLMs. Alongside this instructional advancement, we have also
optimized the visual feature extraction modules in MMLMs, further augmenting
their responsiveness to textual content. Our comprehensive experiments on
MMLMs, based on FlanT5 and Vicuna, show that VisLingInstruct significantly
improves zero-shot performance in visual multi-modal tasks. Notably, it
achieves a 13.1% and 9% increase in accuracy over the prior state-of-the-art on
the TextVQA and HatefulMemes datasets. Our main code is available at
https://github.com/Zhudongsheng75/VisLingInstruct.
