  Referring expression comprehension (REC) is a vision-language task to locate
a target object in an image based on a language expression. Fully fine-tuning
general-purpose pre-trained models for REC yields impressive performance but
becomes increasingly costly. Parameter-efficient transfer learning (PETL)
methods have shown strong performance with fewer tunable parameters. However,
applying PETL to REC faces two challenges: (1) insufficient interaction between
pre-trained vision and language encoders, and (2) high GPU memory usage due to
gradients passing through both heavy encoders. To address these issues, we
present M$^2$IST: Multi-Modal Interactive Side-Tuning with M$^3$ISAs: Mixture
of Multi-Modal Interactive Side-Adapters. During fine-tuning, we keep the
pre-trained vision and language encoders fixed and update M$^3$ISAs on side
networks to establish connections between them, thereby achieving parameter-
and memory-efficient tuning for REC. Empirical results on three benchmarks show
M$^2$IST achieves the best performance-parameter-memory trade-off compared to
full fine-tuning and other PETL methods, with only 3.14M tunable parameters
(2.11% of full fine-tuning) and 15.44GB GPU memory usage (39.61% of full
fine-tuning). Source code will soon be publicly available.
