论文笔记(1)Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning

Federated learning aims to train models collaboratively across different clients without sharing data for privacy considerations.
联邦学习旨在(aims to)跨不同客户端协作(collaboratively)训练模型，出于隐私考虑(for privacy considerations)不需要共享数据。
However, one major challenge for this learning paradigm is the data heterogeneity problem, which refers to the discrepancies between the local data distributions among various clients.
然而，这种学习范式的一个主要挑战是数据异质性问题，它(which)指的是(refers to)不同客户之间本地数据分布之间的差异。
To tackle this problem, we first study how data heterogeneity affects the representations of the globally aggregated models.
为了解决(tackle)这个问题，我们首先研究了数据异构性如何影响全局聚合模型(the globally aggregated models)的表示。
Interestingly, we find that heterogeneous data results in the global model suffering from severe dimensional collapse, in which representations tend to reside in a lower-dimensional space instead of the ambient space.
有趣的是，我们发现异构数据导致全局模型遭受严重的维度崩溃，其中表示往往位于低维空间而不是环境空间(the ambient space)。
Moreover, we observe a similar phenomenon on models locally trained on each client and deduce that the dimensional collapse on the global model is inherited from local models.
此外，我们在每个客户端上本地训练的模型上观察到类似的现象，并推断(deduce)全局模型上的维度崩溃(dimensional collapse)是从局部模型继承(inherited)来的。
In addition, we theoretically analyze the gradient flow dynamics to shed light on how data heterogeneity result in dimensional collapse for local models.
另外，我们在理论上分析了梯度流动力学，以阐明数据异质性如何导致局部模型的维数崩溃。
To remedy this problem caused by the data heterogeneity, we propose FEDDECORR, a novel method that can effectively mitigate dimensional collapse in federated learning.
为了解决(remedy)数据异构性带来的问题，我们提出了FEDDECORR，这是一种新的方法，可以有效地缓解联邦学习中的维度崩溃。
Specifically, FEDDECORR applies a regularization term during local training that encourages different dimensions of representations to be uncorrelated.
具体而言，FEDDECORR在局部训练期间应用正则化项，该项鼓励表示的不同维度不相关。
FEDDECORR, which is implementation-friendly and computationally-efficient, yields consistent improvements over baselines on standard benchmark datasets.