  Different people speak with diverse personalized speaking styles. Although
existing one-shot talking head methods have made significant progress in lip
sync, natural facial expressions, and stable head motions, they still cannot
generate diverse speaking styles in the final talking head videos. To tackle
this problem, we propose a one-shot style-controllable talking face generation
framework. In a nutshell, we aim to attain a speaking style from an arbitrary
reference speaking video and then drive the one-shot portrait to speak with the
reference speaking style and another piece of audio. Specifically, we first
develop a style encoder to extract dynamic facial motion patterns of a style
reference video and then encode them into a style code. Afterward, we introduce
a style-controllable decoder to synthesize stylized facial animations from the
speech content and style code. In order to integrate the reference speaking
style into generated videos, we design a style-aware adaptive transformer,
which enables the encoded style code to adjust the weights of the feed-forward
layers accordingly. Thanks to the style-aware adaptation mechanism, the
reference speaking style can be better embedded into synthesized videos during
decoding. Extensive experiments demonstrate that our method is capable of
generating talking head videos with diverse speaking styles from only one
portrait image and an audio clip while achieving authentic visual effects.
Project Page: https://github.com/FuxiVirtualHuman/styletalk.
