  We explore active audio-visual separation for dynamic sound sources, where an
embodied agent moves intelligently in a 3D environment to continuously isolate
the time-varying audio stream being emitted by an object of interest. The agent
hears a mixed stream of multiple audio sources (e.g., multiple people
conversing and a band playing music at a noisy party). Given a limited time
budget, it needs to extract the target sound accurately at every step using
egocentric audio-visual observations. We propose a reinforcement learning agent
equipped with a novel transformer memory that learns motion policies to control
its camera and microphone to recover the dynamic target audio, using
self-attention to make high-quality estimates for current timesteps and also
simultaneously improve its past estimates. Using highly realistic acoustic
SoundSpaces simulations in real-world scanned Matterport3D environments, we
show that our model is able to learn efficient behavior to carry out continuous
separation of a dynamic audio target. Project:
https://vision.cs.utexas.edu/projects/active-av-dynamic-separation/.
