Multimodal Emotion Recognition for Conversational Systems in Continuous Affective Space

Recognizing human emotions in a continuous affective space is challenging due to their dynamic, multimodal, and context-dependent nature. While the Valence-Arousal-Dominance (VAD) model offers a more precise representation than discrete classification, most continuous emotion recognition (CER) re-search remains unimodal (speech or text), limiting robustness. Speech-text fusion has improved performance, but bimodal approaches still fail to capture the full emotional complexity, emphasizing the need for a fully multimodal solution. To address this gap, we propose a fully multimodal model for CER in conver-sations, extending CORECT, originally designed for multimodal discrete emotion recognition (DER), to the continuous domain. By leveraging relational and temporal dependencies across audio, visual, and textual modalities, CORECT achieves state-of-the-art performance, with an average Concordance Correlation Coefficient (CCC) of 0.774, reflecting an approximately 20% relative improvement over speech-text fusion models (CCC = 0.648).