[综述笔记]Diagnosis of brain diseases in fusion of neuroimaging modalities using deep learning: A review

（2）我已经厌倦了...为什么大多综述都在平平无奇地介绍基础模型....虽然这是很常规的操作但是...阻碍了科学的进步与发展...果然还是tpami天下第一...仍然笼罩在Graph Neural Networks in Network Neuroscience的光环里...年少时不能遇见太惊艳的论文...如果大家都只是简单而没有创新地介绍CNN、GAN、RNN，那一区和三区有什么区别...

（3）我现在真的觉得整合某某种类文章的数量真的是...一张图还行，几张都是就真的水论文了

1.2. 论文总结图

2. 论文逐段精读

2.1. Abstract

①Fusion of multi image helps to more accurate diagnosis

2.2. Introduction

①Introducing brain, neuropsychiatric disorders (introduction and list diseases), neuroimaging (broad introduction and specific explain), imaging fusion

②⭐Structrual imaging: CT, sMRI,DTI; functional imaging: PET,SPECT, fMRI and MEG

	Advantage	Disadvantage
CT	high resolution, short scanning time, high penetration depth	limited tissue characterization, exposure to X-rays, expensive and high radiation
PET	high sensitivity, high penetration depth, and the possibility of imaging physiological and biochemical phenomena	limited resolution, radiation, high cost, motion artifacts, problems interpreting images, and radioactive material usage limitations
SPECT	provide real 3D information, sensitivity, higher penetration depth, and the lack of background in the images	blurring effects, limited resolution, radiation, and high cost
MRI	various structural and functional modalities, high-contrast brain tissue presentation, high-resolution (1 mm cubic voxels), adequate signal-to-noise ratio, and no radiation exposure	long recording time, complex analysis of images

③什么？？？神经成像模式是完美的？？？what

④Image fusion in:

input	raw image fusion or multi-resolution transformations
the layer level	fusion in feature space
the decision level	fusion in classifier

peripheral adj. 次要的，附带的；外围的，周边的；（计算机设备）外围的；（剖）体表的；周围神经系统的，末梢的 n. （计算机的）外部设备

epigenetic adj. 后生的；外成的；渐成说的；表观遗传的

metabolism n. 新陈代谢

2.3. Diagnosis of brain diseases using fusion neuroimaging modalities and AI methods

①Year:2016-2022

②Contributions: review all the DL methods used in analysing multi-MRI; structure is input/layer/decision

③Published fusion methods:

2.4. Search strategy based on PRISMA guidelines

①Providing indexing criteria:

2.5. Brain diseases detection using fusion of neuroimaging modalities

①Modalities applied for different disorders:

2.6. Deep learning models

2.6.1. Convolutional neural network

（1）Pretrained models

（2）U-net models

（3）FCN models

（4）GAN models

（5）Autoencoder models

（6）RNN models

2.7. Fusion levels

①Overall figure of fusion:

2.7.1. Pixel/ input-level fusion

①Pixel-level research collection:

②Pixel-level fusion techniques: multi-scale decomposition (MSD), sparse representation, component substitution, and hybrid model-based methods

③Architectures of pixel-level fusion: imaging, registration, preprocessing, fusion, post-processing, and display

④Pixel-level fusion（没什么好看的就是在输入的时候就融合了，后面就是正常的DL）:

2.7.2. Feature/ layer-level fusion

①Feature-level research collection:

②Feature-level fusion steps（就是单独走线然后合在一起走MLP）:

2.7.3. Decision-level fusion

①Decision-level research collection:

②Decision level fusion flow（和feature-level几乎一模一样不一样的是FC前面的箭头，这里先融合再FC了）:

③Different approaches of decision level fusion: 作者说第一种是每组特征去不同的分类器，那肯定不是上面这个图；然后第二种作者说是用两种不同的模型分支（②图中model 1和model 2），但是明明都不同了，图上还非要画得一模一样...甚至和上一节feature-level也一样...画都画了不能多下点功夫吗？而且为什么不把两种一起画出来呢？

2.7.4. Hybrid-level fusion

①4 situation of hybrid-level fusion（一会儿feature level一会儿layer level能不能统一一下啊？？？）

pixel-level + layer-level

pixel-level + decision-level

feature-level + decision-level

pixel-level + layer-level + decision-level

②Hybrid-level research collection:

2.8. Challenges

①Challenges in neuroimaging: dataset limitations, imbalanced data, DL models, and hardware resources

2.8.1. Limited datasets

①Lack of sample size

2.8.2. Data privacy

①New data privacy policy made（真真？患者数据明明都是被匿名拿去使用了，根本没有过问过吧？这么说起来真是搞笑，别装圣母好吧）

2.8.3. Class imbalance

①The number of HC usually more than diseased, the classification result may sensitive to HC more

2.8.4. Deep learning

①Computing cost

2.8.5. Hardware resources

①怎么和上面一段这么相似啊？？

②凭什么开口就来实现4D不行？我都读过4D的：[论文精读]Characterizing functional brain networks via Spatio-Temporal Attention 4D Convolutional Neural_gambling language motor relational-CSDN博客

甚至还有5D的：

[论文精读]A novel 5D brain parcellation approach based on spatio-temporal encoding of resting fMRI data-CSDN博客

（1）Cloud computing

①Helps to compute

②Disadvantages: expensive, internet required

2.9. Discussion

2.9.1. Comparison of this work with other review papers

①Multi-modal review:

②Comparison figue in content with related revie works:

2.9.2. Fusion level

①为什么又要介绍四种融合啊？？？

②high-level fusions ensure the model’s training and convergence, and low-level fusions integrate information from different modality

③Number of papers with each fusion methods:

2.9.3. Fusion levels in years

①为什么同一个表和图要反复介绍好几次？？

②Accepted fusion level papers:

2.9.4. Types of application

①Applications of multi-modality neuroimaging analysis

2.9.5. Diagnosis of brain diseases using fusion of neuroimaging modalities

①Disease diagnosis（哥们儿这么多个环你大可做一张表全涵盖了，非要这么水空间吗这个篇数真的非常重要吗？）:

2.9.6. Types of neuroimaging modalities

①Number of modality used:

2.9.7. Deep learning models

①Number of backbone used:

2.9.8. Classifiers

①Classifier statistics:

2.10. Future work

2.10.1. Datasets

①Multi-modality data required

2.10.2. Data augmentation methods

①Parameter adjusting

②Data augmentation: transformations, including rotation, translation, scaling, flipping, distortion, and adding noise

2.10.3. Data imbalance

①Resampling methods: under-sampling the negative class, up-sampling the negative class, and synthetic minority over-sampling technique (SMOTE)

②Disadvantage: detelection of important data; redundant data added

2.10.4. DL models

（1）Deep attention mechanism

（2）Transformer models

（3）Graph CNN models

（4）Semi-supervised learning (SSL)

（5）Deep models with uncertainty

2.10.5. Explainability and interpretability

①Using protocols that demonstrate credibility for unified evaluation

2.10.6. Fuzzy deep learning

①Introducing fuzzy techniques

2.10.7. Hardware resources

①Cloud computing

②deep compact-size CNNs

2.11. Conclusion and findings

①sMRI is suitable for brain disease diagnosis and fMRI is suitable for mental disorder diagnosis

②Concluding each section

3. 知识补充

3.1. distributional drifts

数据集的distributional drifts，也称为数据漂移或分布漂移，是机器学习中的一个重要概念。它主要描述的是随着时间的推移，数据集的分布（即数据的统计属性）发生了显著的变化。这种变化可能会影响机器学习模型的性能，因为模型是基于历史数据（即训练数据）进行学习和训练的，而当新数据（即测试或实际使用中的数据）的分布与训练数据的分布不同时，模型的预测能力可能会受到影响。

数据漂移可以分为两种主要类型：

协变量漂移（Covariate Shift）：这种漂移发生在输入数据的分布（即X的分布）随时间发生变化时。换句话说，训练数据（P_train(X)）和测试数据（P_test(X)）的输入分布不同，但给定输入X下输出Y的条件分布（即P_train(Y|X)和P_test(Y|X)）是相同的。这种漂移可能是由于外部因素引起的，例如市场趋势的变化、用户行为的改变或者数据收集过程的变更

概念漂移（Concept Drift）：与协变量漂移不同，概念漂移发生在条件分布（即P_train(Y|X)和P_test(Y|X)）发生变化时，即使输入数据的分布（P_train(X)和P_test(X)）保持不变。这意味着模型需要学习的映射关系已经发生了改变，这可能是由于业务逻辑的变化、新的类别出现或者其他因素导致的。

处理数据漂移对于保持机器学习模型的准确性和性能至关重要。一种常见的策略是定期重新训练模型，以使其能够适应新的数据分布。此外，还可以采用一些专门的漂移检测方法和技术来监控数据分布的变化，并在必要时采取相应的措施来更新或调整模型。

4. Reference

Shoeibi, A. et al. (2023) 'Diagnosis of brain diseases in fusion of neuroimaging modalities using deep learning: A review', Information Fusion, 93: 85-117. doi: Redirecting