  Context-aware emotion recognition (CAER) has recently boosted the practical
applications of affective computing techniques in unconstrained environments.
Mainstream CAER methods invariably extract ensemble representations from
diverse contexts and subject-centred characteristics to perceive the target
person's emotional state. Despite advancements, the biggest challenge remains
due to context bias interference. The harmful bias forces the models to rely on
spurious correlations between background contexts and emotion labels in
likelihood estimation, causing severe performance bottlenecks and confounding
valuable context priors. In this paper, we propose a counterfactual emotion
inference (CLEF) framework to address the above issue. Specifically, we first
formulate a generalized causal graph to decouple the causal relationships among
the variables in CAER. Following the causal graph, CLEF introduces a
non-invasive context branch to capture the adverse direct effect caused by the
context bias. During the inference, we eliminate the direct context effect from
the total causal effect by comparing factual and counterfactual outcomes,
resulting in bias mitigation and robust prediction. As a model-agnostic
framework, CLEF can be readily integrated into existing methods, bringing
consistent performance gains.
