  Recent neural talking radiance field methods have shown great success in
photorealistic audio-driven talking face synthesis. In this paper, we propose a
novel interactive framework that utilizes human instructions to edit such
implicit neural representations to achieve real-time personalized talking face
generation. Given a short speech video, we first build an efficient talking
radiance field, and then apply the latest conditional diffusion model for image
editing based on the given instructions and guiding implicit representation
optimization towards the editing target. To ensure audio-lip synchronization
during the editing process, we propose an iterative dataset updating strategy
and utilize a lip-edge loss to constrain changes in the lip region. We also
introduce a lightweight refinement network for complementing image details and
achieving controllable detail generation in the final rendered image. Our
method also enables real-time rendering at up to 30FPS on consumer hardware.
Multiple metrics and user verification show that our approach provides a
significant improvement in rendering quality compared to state-of-the-art
methods.
