General Multi-modal Image Fusion Transformer Network

Abstract

In the field of image fusion, images obtained from multiple different sensors are fused into one image that contains more complementary information and fewer redundant features to produce a single image with enhanced information. In order to be accurate and concise for different Fusion tasks, this paper proposes a General end-to-end Multi-modal image fusion Transformer Network (GMTN). A two-branch feature extraction module is designed, which integrates the self-attention mechanism of the improved convolutional neural network CNN and Transformer to extract the short-range features and remote dependencies of the image respectively, and take into account various information of the fused image in a more comprehensive way. A large number of experimental results show that the proposed method achieves the same or even better performance than the existing image fusion on a variety of multi-modal medical image data sets, and also achieves good results on infrared and visible light, multi-focus and multi-exposure image fusion tasks.