Transformer With Decoupled Self-Attention Regularization for Age-Unbiased Facial Expression Recognition

Abstract

Facial expression recognition (FER) is a challenging task that involves inferring human emotions from facial images entangled with various attributes. Although FER models based on Transformer have recently reported impressive performance, two major types of bias continue to degrade accuracy—namely, age-related facial attributes such as wrinkles and skin texture, which introduce confusion in recognizing elderly emotions, and imbalanced training data, which limits the model's ability to generalize across age groups. This paper proposes a novel bias-mitigation method that decouples and regularizes age- and emotion-related components within the self-attention mechanism of Transformer. The proposed method separately regularizes the value vectors that encode texture information correlated with age, and the query-key matrices that focus on facial landmarks crucial for emotion recognition. It encourages intra-class compactness of facial landmarks for emotion representation while minimizing age interference in features unrelated to emotion. To suppress age information and preserve emotional features, an age discriminator is employed to guide the value vectors in eliminating age cues, while an emotion classifier restores discriminative information for classification. In addition, the proposed method leverages triplet learning on the query-key space to enhance age-invariant emotion separation. Experiments conducted on four widely used FER benchmarks demonstrate that our method notably reduces age-related bias while maintaining or exceeding state-of-the-art performance.