arxiv:2503.08307

^RFLAV: Rolling Flow matching for infinite Audio Video generation

Published on Mar 11

· Submitted by

Alex Ergasti on Mar 12

Upvote

Authors:

Alex Ergasti ,

Abstract

A novel transformer-based architecture addresses challenges in audio-video generation by effectively aligning audio and visual modalities using a lightweight temporal fusion module.

AI-generated summary

Joint audio-video (AV) generation is still a significant challenge in generative AI, primarily due to three critical requirements: quality of the generated samples, seamless multimodal synchronization and temporal coherence, with audio tracks that match the visual data and vice versa, and limitless video duration. In this paper, we present , a novel transformer-based architecture that addresses all the key challenges of AV generation. We explore three distinct cross modality interaction modules, with our lightweight temporal fusion module emerging as the most effective and computationally efficient approach for aligning audio and visual modalities. Our experimental results demonstrate that outperforms existing state-of-the-art models in multimodal AV generation tasks. Our code and checkpoints are available at https://github.com/ErgastiAlex/R-FLAV.

View arXiv page View PDF GitHub 9 Add to collection

Community

MaverickAlex

Paper author Paper submitter Mar 12

Joint audio-video (AV) generation is still a significant challenge in generative AI, primarily due to three critical requirements: quality of the generated samples, seamless multimodal synchronization and temporal coherence, with audio tracks that match the visual data and vice versa, and limitless video duration. In this paper, we present R-FLAV, a novel transformer-based architecture that addresses all the key challenges of AV generation. We explore three distinct cross modality interaction modules, with our lightweight temporal fusion module emerging as the most effective and computationally efficient approach for aligning audio and visual modalities. Our experimental results demonstrate that R-FLAV, outperforms existing state-of-the-art models in multimodal AV generation tasks.