Multimodal Emotion Recognition Based on Deep Learning

Abstract

In recent years, multitask learning-based joint analysis of multiple emotions has emerged as a significant research topic in natural language processing and artificial intelligence. This approach aims to identify multiple emotion categories expressed in discourse by integrating multimodal information and leveraging shared knowledge across related tasks. Sentiment analysis, emotion recognition, and sarcasm detection constitute three closely interconnected tasks in affective computing. This paper focuses on these three tasks - sentiment analysis, emotion recognition, and sarcasm detection - while addressing current challenges in their research. The specific work includes the following three aspects: (1) Due to the limitations of datasets in the development of current Chinese multi-task learning models, this paper establishes a Chinese multi-task multi-modal dialogue emotion corpus to support the development of multi-task multi-modal sentiment analysis. The dataset is annotated with multiple task labels (such as sentiment, emotion, sarcasm, humor, etc.) and, for the first time, manually annotates the correlation between sentiment and emotion, as well as sarcasm and humor. Through scientific evaluation and analysis, it is demonstrated that the dataset possesses high quality and representativeness. (2) Based on the constructed dataset, this paper primarily considers three aspects: context interaction, multi-modal feature fusion, and multi-task learning, and proposes a multi-modal sentiment analysis model based on multi-task learning. Through experimental evaluation, the effectiveness of the model is demonstrated. (3) In response to the model proposed in (2), which fails to consider the interrelationships between tasks, this paper presents a multi-task learning model based on soft parameter sharing to learn the commonalities and differences between different tasks. Experimental results comparing with other advanced baselines demonstrate the innovation and efficiency of the proposed method.