Adaptive Multitask Learning With Uncertainty‐Aware Loss Balancing for Multimodal Emotion Recognition

Multimodal emotion recognition (MER) systems must balance competing learning objectives, namely classification, feature disentanglement and cross‐modal alignment, yet existing frameworks rely on manually tuned loss weights, limiting adaptability across diverse input conditions. To remedy this, we propose MER‐MTL, a multitask learning framework that integrates modality reconstruction as auxiliary tasks and adopts the homoscedastic uncertainty strategy to learn loss weights end‐to‐end. Experiments on CMU‐MOSI show that MER‐MTL achieves 43.29% seven‐class accuracy under aligned conditions, a statistically significant improvement over our DMD reimplementation (37.50%, = 0.009). Under unaligned conditions, MER‐MTL maintains its advantage (+5.22%, = 0.003). The uncertainty‐aware mechanism also reduces prediction variance by up to 48% across random seeds, demonstrating improved training stability. The source code is available at https://github.com/tangtangma/MER‐MTL .