Enhancing Multimodal Continuous Emotion Recognition with Self-Supervised Speech Representations

Continuous emotion recognition (CER) aims to model affective states in a fine-grained dimensional space rather than through rigid categorical labels. While multimodal conversational architectures effectively exploit contextual, temporal, and cross-modal dependencies, their acoustic branch traditionally relies on handcrafted descriptors that may fail to capture high-level paralinguistic information. In this work, we investigate the integration of selfsupervised speech representations - HuBERT Large and WavLM Large - into a graph-based multimodal conversational framework, namely CORECT (COnversation understanding using RElational temporal graph neural networks with auxiliary Cross-modality interacTion), for valencearousal-dominance prediction. Performance is measured using the Concordance Correlation Coefficient (CCC), which jointly captures correlation and scale consistency between predicted and reference values. Experiments on the IEMOCAP corpus show that the proposed WavLMbased framework reaches a Mean CCC of 0.7005, outperforming both the handcrafted-feature CORECT model (+4.7% relative) and prior bimodal late-fusion baselines (+8.1% relative). Further analysis reveals that valence primarily benefits from cross-modal semantic integration, while arousal is mainly driven by acoustic content. These results highlight the relevance of self-supervised speech representation learning for robust continuous emotion recognition in conversational systems.