  Although massive pre-trained vision-language models like CLIP show impressive
generalization capabilities for many tasks, still it often remains necessary to
fine-tune them for improved performance on specific datasets. When doing so, it
is desirable that updating the model is fast and that the model does not lose
its capabilities on data outside of the dataset, as is often the case with
classical fine-tuning approaches. In this work we suggest a lightweight
adapter, that only updates the models predictions close to seen datapoints. We
demonstrate the effectiveness and speed of this relatively simple approach in
the context of few-shot learning, where our results both on classes seen and
unseen during training are comparable with or improve on the state of the art.
