【论文研读】Similarity of Neural Network Representations Revisited (ICML2019)

最新推荐文章于 2024-06-08 09:50:29 发布

Always_ease

最新推荐文章于 2024-06-08 09:50:29 发布

阅读量1.9k

点赞数 2

分类专栏：论文研读文章标签：神经网络 ICML Similarity

本文链接：https://blog.csdn.net/Always_ease/article/details/110918015

版权

该论文（ICML2019）研究了神经网络表示的相似性，关注相似性指数的不变性特性。作者发现早期层相对于后期层更能学习到相似的表示。文章对比了不同相似性结构，如内积基的相似性、Hilbert-Schmidt独立性准则（HSIC）和中心化核对齐（CKA）。CKA被证明在揭示层间对应关系方面最为有效，尤其在不同网络架构中。

摘要由CSDN通过智能技术生成

Title: Similarity of Neural Network Representations Revisited (ICML2019)

Author：Simon Kornblith ...(Long Beach, California)

Aim:

invariance properties of similarity indexes 分为三个方面

Comparing Similarity Structures

Results

Conclusion and Future Work

Aim:

one can first measure the similarity between every pair of examples in each representation separately, and then compare the similarity structures.

invariance properties of similarity indexes 分为三个方面

1. Invariance to Invertible Linear Transformation

定义：A similarity index is invariant to invertible linear transformation if s(X, Y ) = s(XA, Y B) for any full rank A and B.

Key sentence：

We demonstrate that early layers, but not later layers, learn similar representations on different datasets.
Invariance to invertible linear transformation implies that the scale of directions in activation space is irrelevant.
Neural networks trained from different random initializations develop representations with similar large principal components 不同初始化得到的主要参数是相似的。因此基于主要参数得到的不同网络间的相似性度量（如欧氏距离）的相似的。A similarity index that is invariant to invertible linear transformation ignores this aspect of the representation, and assigns the same score to networks that match only in large principal components or networks that match only in small principal components.

2. Invariance to Ort