图像融合论文速读：MUFusion: A general unsupervised image fusion network based on memory unit

@article{cheng2023mufusion,
title={MUFusion: A general unsupervised image fusion network based on memory unit},
author={Cheng, Chunyang and Xu, Tianyang and Wu, Xiao-Jun},
journal={Information Fusion},
volume={92},
pages={80–92},
year={2023},
publisher={Elsevier}
}

---

论文级别：SCI A1 TOP

影响因子：18.6

📖[论文下载地址]
💽[代码下载地址]

---

---

📖论文解读

该文提出了一种基于新的记忆单元结构的自进化训练的【端到端】的【通用】图像融合模型MUFusion，可以处理多模态图像融合（IVIF、医学）及数字摄影图像融合（多聚焦、多曝光）任务。

🔑关键词

Image fusion 图像融合
Memory unit 记忆单元
General model 通用模型
Deep learning 深度学习
Unsupervised learning 无监督学习

💭核心思想

这篇文章，只要理解两点就通透了
分别是【记忆单元】和【自适应结构损失权重】

利用训练过程中获得的【中间融合结果】作为辅助信息，监督融合后的图像。既可以从原始输入图像中学习，也可以从网络本身的中间输出中受益。
下图就是作者提出基于记忆单元的方法，x和y分别为输入图像和融合图像。首先生成SA和SB这两个结构相似性变量，然后通过softmax计算自适应的权值p1和p2，这两个权值是用来控制内容损失和记忆损失比例关系的

训练时，在每次迭代中，将源图像送入深度网络，产生相应的融合结果。使用包含内容损失和记忆损失两项的损失函数来监督该网络。
同时，将前一epoch得到的模型参数，连同当前的输入输出一起输入到存储单元中。
此外，利用预训练的VGG-19网络提取源图像的深度特征，然后使用L1范式获取活动级别图，通过执行“选择最大”操作获得显著掩膜。

🎖️本文贡献

• 设计了一种新的记忆单元，利用训练过程中的中间结果作为辅助监督
• 使用活动等级图度量输入图像每个像素的重要程度
• 设计了由内容损失和记忆损失组成的损失函数

🪅相关背景知识

• 深度学习
• 神经网络
• 图像融合

扩展学习
[什么是图像融合？（一看就通，通俗易懂）]

🪢网络结构

作者提出的网络结构如下所示。

MUFusion具体的网络结构如表1所示。

📉损失函数

损失函数=内容损失+记忆损失

内容损失=像素损失+结构相似性损失+梯度损失


再具体的细节可以去原文学习，总之只要知道𝑤1和𝑤2是两个二元显著区掩膜就可以了。

记忆损失是基于源图像的活动级别映射计算，可以约束输出图像以包含特定信息，根据模型之前的输出得到记忆损失，可迫使网络产生更高质量的融合结果

🔢数据集

• IVIF
  TNO,ROADSCENE

图像融合数据集链接
[图像融合常用数据集整理]

🎢训练设置

🔬实验

📏评价指标

• IVIF
  VIF,EN,MI,EI,QABF

扩展学习
[图像融合定量指标分析]

🥅Baseline

• IVIF
  CBF，MDLatLRR ，DenseFuse，PMGI，U2Fusion， PIAFusion ， RFN-Nest.

✨✨✨扩展学习✨✨✨
✨✨✨强烈推荐必看博客[图像融合论文baseline及其网络模型]✨✨✨

🔬实验结果

更多实验结果及分析可以查看原文：
📖[论文下载地址]

🧷总结体会

【记忆单元】这个创行点非常有想法，借鉴了RNN的思想实现IF任务

---

🚀传送门

📑图像融合相关论文阅读笔记

📑[(TLGAN)Boosting target-level infrared and visible image fusion with regional information coordination]
📑[ReFusion: Learning Image Fusion from Reconstruction with Learnable Loss via Meta-Learning]
📑[YDTR: Infrared and Visible Image Fusion via Y-Shape Dynamic Transformer]
📑[CS2Fusion: Contrastive learning for Self-Supervised infrared and visible image fusion by estimating feature compensation map]
📑[CrossFuse: A novel cross attention mechanism based infrared and visible image fusion approach]
📑[(DIF-Net)Unsupervised Deep Image Fusion With Structure Tensor Representations]
📑[(MURF: Mutually Reinforcing Multi-Modal Image Registration and Fusion]
📑[(A Deep Learning Framework for Infrared and Visible Image Fusion Without Strict Registration]
📑[(APWNet)Real-time infrared and visible image fusion network using adaptive pixel weighting strategy]
📑[Dif-fusion: Towards high color fidelity in infrared and visible image fusion with diffusion models]
📑[Coconet: Coupled contrastive learning network with multi-level feature ensemble for multi-modality image fusion]
📑[LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images]
📑[(DeFusion)Fusion from decomposition: A self-supervised decomposition approach for image fusion]
📑[ReCoNet: Recurrent Correction Network for Fast and Efficient Multi-modality Image Fusion]
📑[RFN-Nest: An end-to-end resid- ual fusion network for infrared and visible images]
📑[SwinFuse: A Residual Swin Transformer Fusion Network for Infrared and Visible Images]
📑[SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer]
📑[(MFEIF)Learning a Deep Multi-Scale Feature Ensemble and an Edge-Attention Guidance for Image Fusion]
📑[DenseFuse: A fusion approach to infrared and visible images]
📑[DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pair]
📑[GANMcC: A Generative Adversarial Network With Multiclassification Constraints for IVIF]
📑[DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion]
📑[IFCNN: A general image fusion framework based on convolutional neural network]
📑[(PMGI) Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity]
📑[SDNet: A Versatile Squeeze-and-Decomposition Network for Real-Time Image Fusion]
📑[DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion]
📑[FusionGAN: A generative adversarial network for infrared and visible image fusion]
📑[PIAFusion: A progressive infrared and visible image fusion network based on illumination aw]
📑[CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion]
📑[U2Fusion: A Unified Unsupervised Image Fusion Network]
📑综述[Visible and Infrared Image Fusion Using Deep Learning]

📚图像融合论文baseline总结

📚[图像融合论文baseline及其网络模型]

📑其他论文

📑[3D目标检测综述：Multi-Modal 3D Object Detection in Autonomous Driving：A Survey]

🎈其他总结

🎈[CVPR2023、ICCV2023论文题目汇总及词频统计]

✨精品文章总结

✨[图像融合论文及代码整理最全大合集]
✨[图像融合常用数据集整理]

🌻【如侵权请私信我删除】

如有疑问可联系：420269520@qq.com;
码字不易，【关注，收藏，点赞】一键三连是我持续更新的动力，祝各位早发paper，顺利毕业~