  While few-shot learning as a transfer learning paradigm has gained
significant traction for scenarios with limited data, it has primarily been
explored in the context of building unimodal and unilingual models.
Furthermore, a significant part of the existing literature in the domain of
few-shot multitask learning perform in-context learning which requires manually
generated prompts as the input, yielding varying outcomes depending on the
level of manual prompt-engineering. In addition, in-context learning suffers
from substantial computational, memory, and storage costs which eventually
leads to high inference latency because it involves running all of the prompt's
examples through the model every time a prediction is made. In contrast,
methods based on the transfer learning via the fine-tuning paradigm avoid the
aforementioned issues at a one-time cost of fine-tuning weights on a per-task
basis. However, such methods lack exposure to few-shot multimodal multitask
learning. In this paper, we propose few-shot learning for a multimodal
multitask multilingual (FM3) setting by adapting pre-trained vision and
language models using task-specific hypernetworks and contrastively fine-tuning
them to enable few-shot learning. FM3's architecture combines the best of both
worlds of in-context and fine-tuning based learning and consists of three major
components: (i) multimodal contrastive fine-tuning to enable few-shot learning,
(ii) hypernetwork task adaptation to perform multitask learning, and (iii)
task-specific output heads to cater to a plethora of diverse tasks. FM3 learns
the most prominent tasks in the vision and language domains along with their
intersections, namely visual entailment (VE), visual question answering (VQA),
and natural language understanding (NLU) tasks such as neural entity
recognition (NER) and the GLUE benchmark including QNLI, MNLI, QQP, and SST-2.
