Attention-based Dual-Branch Network for Micro-Expression Recognition with Global-Local Feature Fusion

Micro-expression(ME) is an uncontrollable muscle movement that appears on the face when people try to hide or inhibit their real emotions, which has the problems of short duration, small movement amplitude and uneven distribution. In order to solve the problem of localization and asymmetry in the distribution of ME features, this paper proposes a new two-branch attention network to recognize MEs, which utilizes the attention mechanism to capture global and local ME features. The network is mainly divided into three parts: data preprocessing, ME feature learning and feature fusion classification. First, the data preprocessing first extracts the ME optical flow features and then divides them into four regions, which are used as inputs to the two-branch network respectively. Second, the two-branch network uses the Inception-MSFE global multi-scale feature extraction network incorporating the attention mechanism (CBAM) and the Swin Transformer-based local feature extraction network for feature learning, respectively. Finally, MEs are predicted by fusing global and local features of MEs. Experimental validation is carried out on three datasets, CASME II, SAMM, and SMIC, which proves that Acc, UAR, and UF1 are 0.797, 0.702, and 0.698 on the SAMM dataset, respectively; Acc, UAR, and UF1 are 0.734, 0.723, and 0.729 on SMIC, respectively; and on the CASME II dataset, Acc, UAR and UF1 are 0.865, 0.872, and 0.889, respectively, which are competitive with other state-of-the-art methods.