  Current Facial Action Unit (FAU) detection methods generally encounter
difficulties due to the scarcity of labeled video training data and the limited
number of training face IDs, which renders the trained feature extractor
insufficient coverage for modeling the large diversity of inter-person facial
structures and movements. To explicitly address the above challenges, we
propose a novel video-level pre-training scheme by fully exploring the
multi-label property of FAUs in the video as well as the temporal label
consistency. At the heart of our design is a pre-trained video feature
extractor based on the video-masked autoencoder together with a fine-tuning
network that jointly completes the multi-level video FAUs analysis tasks,
\emph{i.e.} integrating both video-level and frame-level FAU detections, thus
dramatically expanding the supervision set from sparse FAUs annotations to ALL
video frames including masked ones. Moreover, we utilize inter-frame and
intra-frame AU pair state matrices as prior knowledge to guide network training
instead of traditional Graph Neural Networks, for better temporal supervision.
Our approach demonstrates substantial enhancement in performance compared to
the existing state-of-the-art methods used in BP4D and DISFA FAUs datasets.
