  Vision language models (VLMs) demonstrate impressive capabilities in visual
question answering and image captioning, acting as a crucial link between
visual and language models. However, existing open-source VLMs heavily rely on
pretrained and frozen vision encoders (such as CLIP). Despite CLIP's robustness
across diverse domains, it still exhibits non-negligible image understanding
errors. These errors propagate to the VLM responses, resulting in sub-optimal
performance. In our work, we propose an efficient and robust method for
updating vision encoders within VLMs. Our approach selectively and locally
updates encoders, leading to substantial performance improvements on data where
previous mistakes occurred, while maintaining overall robustness. Furthermore,
we demonstrate the effectiveness of our method during continual few-shot
updates. Theoretical grounding, generality, and computational efficiency
characterize our approach.
