  In the realm of deep learning, maintaining model robustness against
distribution shifts is critical. This paper investigates test-time adaptation
strategies for vision-language models, with a specific focus on CLIP and its
variants. Through a systematic exploration of prompt-based techniques and
existing test-time adaptation methods, the study aims to enhance the
adaptability and robustness of vision-language models in diverse real-world
scenarios. The investigation includes an analysis of prompt engineering
strategies, such as hand-crafted prompts, prompt ensembles, and prompt learning
techniques. We introduce a vision-text-space ensemble that significantly boosts
the average performance compared to a text-space-only ensemble. Additionally,
our comparative study delves into leveraging existing test-time adaptation
methods originally designed for image classification tasks. Experimental
evaluations conducted across various datasets and model architectures
demonstrate the efficacy of different adaptation strategies. We further give
insights into the importance of updating the vision encoder and whether it is
beneficial to update the text encoder. Code is available at
https://github.com/mariodoebler/test-time-adaptation
