  Generating face image with specific gaze information has attracted
considerable attention. Existing approaches typically input gaze values
directly for face generation, which is unnatural and requires annotated gaze
datasets for training, thereby limiting its application. In this paper, we
present a novel gaze-controllable face generation task. Our approach inputs
textual descriptions that describe human gaze and head behavior and generates
corresponding face images. Our work first introduces a text-of-gaze dataset
containing over 90k text descriptions spanning a dense distribution of gaze and
head poses. We further propose a gaze-controllable text-to-face method. Our
method contains a sketch-conditioned face diffusion module and a model-based
sketch diffusion module. We define a face sketch based on facial landmarks and
eye segmentation map. The face diffusion module generates face images from the
face sketch, and the sketch diffusion module employs a 3D face model to
generate face sketch from text description. Experiments on the FFHQ dataset
show the effectiveness of our method. We will release our dataset and code for
future research.
