【论文研读】Similarity of Neural Network Representations Revisited (ICML2019)

最新推荐文章于 2023-01-31 14:25:09 发布

Always_ease

最新推荐文章于 2023-01-31 14:25:09 发布

阅读量1.6k

点赞数 2

分类专栏：论文研读文章标签：神经网络 ICML Similarity

本文链接：https://blog.csdn.net/Always_ease/article/details/110918015

版权

Author：Simon Kornblith ...(Long Beach, California)

one can first measure the similarity between every pair of examples in each representation separately, and then compare the similarity structures.

1. Invariance to Invertible Linear Transformation

定义：A similarity index is invariant to invertible linear transformation if s(X, Y ) = s(XA, Y B) for any full rank A and B.

Key sentence：

We demonstrate that early layers, but not later layers, learn similar representations on different datasets.
Invariance to invertible linear transformation implies that the scale of directions in activation space is irrelevant.
Neural networks trained from different random initializations develop representations with similar large principal components 不同初始化得到的主要参数是相似的。因此基于主要参数得到的不同网络间的相似性度量（如欧氏距离）的相似的。A similarity index that is invariant to invertible linear transformation ignores this aspect of the representation, and assigns the same score to networks that match only in large principal components or networks that match only in small principal components.

2. Invariance to Orthogonal Transformation

最低0.47元/天解锁文章

关注