  While Vision Transformers (ViTs) have demonstrated remarkable capabilities in
learning representations, their performance is compromised when applied to
unseen domains. Previous methods either engage in prompt learning during the
training phase or modify model parameters at test time through entropy
minimization. The former often overlooks unlabeled target data, while the
latter doesn't fully address domain shifts. In this work, our approach, Optimal
Transport-guided Test-Time Visual Prompting (OT-VP), handles these problems by
leveraging prompt learning at test time to align the target and source domains
without accessing the training process or altering pre-trained model
parameters. This method involves learning a universal visual prompt for the
target domain by optimizing the Optimal Transport distance. With just four
prompt tokens learned, OT-VP achieves a $5.0\%$ and $1.5\%$ increase in
averaged accuracy across single-source and multi-source settings on three
benchmark datasets, which is $1.2\times$ and $1.5\times$ the improvement of the
state-of-the-art method, respectively.
