  We propose Im2Wav, an image guided open-domain audio generation system. Given
an input image or a sequence of images, Im2Wav generates a semantically
relevant sound. Im2Wav is based on two Transformer language models, that
operate over a hierarchical discrete audio representation obtained from a
VQ-VAE based model. We first produce a low-level audio representation using a
language model. Then, we upsample the audio tokens using an additional language
model to generate a high-fidelity audio sample. We use the rich semantics of a
pre-trained CLIP (Contrastive Language-Image Pre-training) embedding as a
visual representation to condition the language model. In addition, to steer
the generation process towards the conditioning image, we apply the
classifier-free guidance method. Results suggest that Im2Wav significantly
outperforms the evaluated baselines in both fidelity and relevance evaluation
metrics. Additionally, we provide an ablation study to better assess the impact
of each of the method components on overall performance. Lastly, to better
evaluate image-to-audio models, we propose an out-of-domain image dataset,
denoted as ImageHear. ImageHear can be used as a benchmark for evaluating
future image-to-audio models. Samples and code can be found inside the
manuscript.
