  Contrastive vision-language models (e.g. CLIP) are typically created by
updating all the parameters of a vision model and language model through
contrastive training. Can such models be created by a small number of parameter
updates to an already-trained language model and vision model? The literature
describes techniques that can create vision-language models by updating a small
number of parameters in a language model, but these require already aligned
visual representations and are non-contrastive, hence unusable for
latency-sensitive applications such as neural search. We explore the
feasibility and benefits of parameter-efficient contrastive vision-language
alignment through transfer learning: creating a model such as CLIP by minimally
updating an already-trained vision and language model. We find that a minimal
set of parameter updates ($<$7%) can achieve the same performance as full-model
training, and updating specific components ($<$1% of parameters) can match 75%
of full-model training. We describe a series of experiments: we show that
existing knowledge is conserved more strongly in parameter-efficient training
and that parameter-efficient scaling scales with model and dataset size. Where
paired-image text data is scarce but strong multilingual language models exist
(e.g. low resource languages), parameter-efficient training is even preferable
to full-model training. Given a fixed compute budget, parameter-efficient
training allows training larger models on the same hardware, achieving
equivalent performance in less time. Parameter-efficient training hence
constitutes an energy-efficient and effective training strategy for contrastive
vision-language models that may be preferable to the full-model training
paradigm for common use cases. Code and weights at
https://github.com/codezakh/LilT.
