  Multimodal few-shot learning is challenging due to the large domain gap
between vision and language modalities. Existing methods are trying to
communicate visual concepts as prompts to frozen language models, but rely on
hand-engineered task induction to reduce the hypothesis space. To make the
whole process learnable, we introduce a multimodal meta-learning approach.
Specifically, our approach decomposes the training of the model into a set of
related multimodal few-shot tasks. We define a meta-mapper network, acting as a
meta-learner, to efficiently bridge frozen large-scale vision and language
models and leverage their already learned capacity. By updating the learnable
parameters only of the meta-mapper, it learns to accrue shared meta-knowledge
among these tasks. Thus, it can rapidly adapt to newly presented samples with
only a few gradient updates. Importantly, it induces the task in a completely
data-driven manner, with no need for a hand-engineered task induction. We
evaluate our approach on recently proposed multimodal few-shot benchmarks,
measuring how rapidly the model can bind novel visual concepts to words and
answer visual questions by observing only a limited set of labeled examples.
The experimental results show that our meta-learning approach outperforms the
baseline across multiple datasets and various training settings while being
computationally more efficient.
