  Facial Action Unit (AU) detection is a crucial task in affective computing
and social robotics as it helps to identify emotions expressed through facial
expressions. Anatomically, there are innumerable correlations between AUs,
which contain rich information and are vital for AU detection. Previous methods
used fixed AU correlations based on expert experience or statistical rules on
specific benchmarks, but it is challenging to comprehensively reflect complex
correlations between AUs via hand-crafted settings. There are alternative
methods that employ a fully connected graph to learn these dependencies
exhaustively. However, these approaches can result in a computational explosion
and high dependency with a large dataset. To address these challenges, this
paper proposes a novel self-adjusting AU-correlation learning (SACL) method
with less computation for AU detection. This method adaptively learns and
updates AU correlation graphs by efficiently leveraging the characteristics of
different levels of AU motion and emotion representation information extracted
in different stages of the network. Moreover, this paper explores the role of
multi-scale learning in correlation information extraction, and design a simple
yet effective multi-scale feature learning (MSFL) method to promote better
performance in AU detection. By integrating AU correlation information with
multi-scale features, the proposed method obtains a more robust feature
representation for the final AU detection. Extensive experiments show that the
proposed method outperforms the state-of-the-art methods on widely used AU
detection benchmark datasets, with only 28.7\% and 12.0\% of the parameters and
FLOPs of the best method, respectively. The code for this method is available
at \url{https://github.com/linuxsino/Self-adjusting-AU}.
