  We propose a modular framework that leverages the expertise of different
foundation models over different modalities and domains in order to perform a
single, complex, multi-modal task, without relying on prompt engineering or
otherwise tailor-made multi-modal training. Our approach enables decentralized
command execution and allows each model to both contribute and benefit from the
expertise of the other models. Our method can be extended to a variety of
foundation models (including audio and vision), above and beyond only language
models, as it does not depend on prompts. We demonstrate our approach on two
tasks. On the well-known task of stylized image captioning, our experiments
show that our approach outperforms semi-supervised state-of-the-art models,
while being zero-shot and avoiding costly training, data collection, and prompt
engineering. We further demonstrate this method on a novel task, audio-aware
image captioning, in which an image and audio are given and the task is to
generate text that describes the image within the context of the provided
audio. Our code is available on GitHub.
