A Survey on Multimodal Emotion Recognition: Integrating Cues for a Deeper Understanding of Affect

Abstract

Multimodal Emotion Recognition (MER) has emerged as a crucial area of research in artificial intelligence and human-computer interaction, aiming to build systems that can understand human affective states by integrating information from various modalities. This review provides a comprehensive overview of the MER landscape, synthesizing insights from foundational and recent literature. We delve into the primary modalities utilized—including visual (facial expressions), acoustic (speech prosody), textual (language content), and physiological signals—and discuss the state-of-the-art deep learning techniques for feature extraction within each. A central focus is placed on multimodal fusion strategies, from early (feature-level) and late (decision-level) fusion to more sophisticated Transformer-based and attention mechanisms that capture complex inter-modal dynamics. We also examine the role of advanced architectures like Multimodal Large Language Models (MLLMs) and techniques such as knowledge distillation for handling real-world challenges like modality missingness. Key benchmark datasets that have propelled the field forward are described. Finally, we outline the persistent challenges, including data scarcity, modality misalignment, and real-world robustness, and propose promising future research directions to advance the development of more accurate, robust, and context-aware affective computing systems.