  Training large foundation models using self-supervised objectives on
unlabeled data, followed by fine-tuning on downstream tasks, has emerged as a
standard procedure. Unfortunately, the efficacy of this approach is often
constrained by both limited fine-tuning compute and scarcity in labeled
downstream data. We introduce Multimodal Attention Merging (MAM), an attempt
that facilitates direct knowledge transfer from attention matrices of models
rooted in high resource modalities, text and images, to those in
resource-constrained domains, speech and audio, employing a zero-shot paradigm.
MAM reduces the relative Word Error Rate (WER) of an Automatic Speech
Recognition (ASR) model by up to 6.70%, and relative classification error of an
Audio Event Classification (AEC) model by 10.63%. In cases where some
data/compute is available, we present Learnable-MAM, a data-driven approach to
merging attention matrices, resulting in a further 2.90% relative reduction in
WER for ASR and 18.42% relative reduction in AEC compared to fine-tuning.
