Understanding Self-Supervised Learning Dynamics without Contrastive Pairs论文笔记

Understanding Self-Supervised Learning Dynamics without Contrastive Pairs论文笔记

Two layer linear model

Points

1. weight decay can balance the predictor and online network weights
2. stop gradient can prevent collapsing
3. stop gradient with no predictor will lead to collapsing.

Model

$W_p\in {\mathbb{R}}^{n_2\times n_2},W\in {\mathbb{R}}^{n_2\times n_1},W_a\in {\mathbb{R}}^{n_2\times n_1},x\in {\mathbb{R}}^{n_1}.f_1=W{x}_1\in {\mathbb{R}}^{n_2},f_{2a}=W_a{x}_2\in {\mathbb{R}}^{n_2}$

$x_1,x_2$ is augmented views.

Loss Function

Gradient

$\eta$: weight decay, $X={\mathbb{E}}_x[\bar{x}{\bar{x}}^T],\bar{x}(x):={\mathbb{E}}_{x^{\prime }\sim p_{aug}(\cdot |x)}[x^{\prime }],X^{\prime }={\mathbb{E}}_x[{\mathbb{V}}_{x^{\prime }|x}[x^{\prime }]]$, $x^{\prime }$ is augmented view of $x$

Proof

1. weight decay can balance the predictor and online network weights (by removing $W_a$)

   

   $${\alpha }_p^{-1}[e^{2{\eta }_{p}t}{W}_p^T{W}_p-W_p^T(0)W_p(0)]=e^{2\eta t}W{W}^T-W(0)W^T(0)$$

2. stop gradient can prevent collapsing

   

   $$\begin{gathered} H(t):=X^{\prime }\otimes (W_p^T{W}_p+I)+X\otimes {\tilde{W}}_p^T{\tilde{W}}_p+\eta I_{n_1{n}_2},W_p=W_p-I_{n_2} \\ H(t):=X^{\prime }\otimes (W_p^T{W}_p+I)+X\otimes (W_p^T{W}_p-2W_p+I)+\eta I_{n_1{n}_2} \end{gathered}$$

3. stop gradient with no predictor will lead to collapsing.

   $W_p=I$

   

Multiple factors analysis

Three Assumptions are made to decouple gradients to scalar.

Assumptions

Assumption 1 (Proportional EMA)

experiments validation

Assumption 2 (Isotropic（各向同性，等方差） data and augmentation).

Data distribution $p(x)$ has zero mean and identity covariance

Augmentation distribution $p_{aug}(\cdot |x)$ has mean $x$ and covariance ${\sigma }^{2}I$.

$X = I, X’ = $${\sigma }^{2}I$

(Previous work)

Assumption 3 (symmetric predictor $W_p$)

$W_p=W_p^T$

• Motivation

  • fixed point ($\dot{W_p}=0$) is symmetric (under some occasions) (particular $\eta ,W{W}^T$)
  • Under Assumption 1 and 2 the asymmetry part ($W_p-W_p^T$) vanishes. (particular $\eta ,\tau$)

• Experiments Phenomena

  • BYOL: symmetric > reg (slightly better)
  • Simsiam: symmetric fails (why?)

  

  $$\bar{\eta }=0.0004,{\alpha }_p=1$$

Conclusion

Under above assumptions, the eigenspace (eigenvectors) of $F$ and $W_p$ gradually align. (experiments validation Fig. 9)

(when $\eta$ is small or zero and $\tau$ is large, the alignment will vanish.)

$F:=WX{W}^T$. Note that $F$ is the correlation matrix of the $W{x}_1$. By Assumption 2, $\mathbb{E}[x]=0$ and F is also the covariance matrix.(?)
$$\begin{gathered} W_p=U{\Lambda }_{W_p}{U}^T,F=U{\Lambda }_F{U}^T,\dot{W_p}=U{G}_1{U}^T,\dot{F}=U{G}_2{U}^T(\dot{U}=0) \\ {\Lambda }_{W_p}=\text{diag}[p_1,p_2,...,p_d],{\Lambda }_F=\text{diag}[s_1,s_2,...,s_d] \end{gathered}$$
带入梯度式子后得到

Analysis on ${\alpha }_p,\eta ,\beta$

(relative predictor learning rate, weight decay, EMA parameter.)

• From Eqn. 11 and Eqn. 12 (remove $\tau$), we have

  

  $$c_j=s_j(0)-{\alpha }_p^{-1}{p}_j^2(0)$$
  和前面的某个式子很像

  • $\eta =0$时，${\alpha }_p$ 太小，$c_j<0$,$s_j\to 0$, might collapse.
  • $\eta =0$时，${\alpha }_p$ 太大，$s_j(+\infty )=s_j(0)$
  • $\eta >0$, $s_j={\alpha }_p^{-1}{p}_j^2$ balance.

• $\eta >0$时，把$s_j={\alpha }_p^{-1}{p}_j^2$带入Eqn. 11得到

  

  $$\begin{gathered} {\dot{p}}_j=p_j{\Delta }_j \\ {\Delta }_j:=p_j[\tau -(1+{\sigma }^2)p_j]-\eta \end{gathered}$$

  令${\dot{p}}_j=0$有，

  

  ${\dot{p}}_j$ 图像如下，${\dot{p}}_{j-}$ unstable，${\dot{p}}_{j0}$ collapse .

  [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-h6cZU5Hz-1626077314449)(图片数据/屏幕截图 2021-06-22 220434.png)]

  • larger $\eta$, smaller $\tau$, lead ${\dot{p}}_{j-}$ to right and the basin of collapse expands.
  • when $\eta >\frac{{\tau }^2}{4(1+{\sigma }^2)}$, collapsing is unavoidable.

• To satisfy eigenspace alignment, below equation has to hold.

  $${\Delta }_j:=p_j[\tau -(1+{\sigma }^2)p_j]-\eta$$

  so larger $\eta$ and ${\alpha }_p$ can loosen the bound.

• Curriculum learning: Initially, $p_j$ and $s_j$ is small, and since $W$ changes rapidly, $\tau$ is also small. When $p_j$ approaches its stable fixed point $p_j^{+}$, then $p_j$ and $s_j$ stop growing, making $\tau$ larger, and set a higher $p_j^{+}$.

• ${\alpha }_p>1,{\eta }_p>{\eta }_s$, can make symmetric $W_p$ work without EMA (Simsiam) ? (更容易满足特征空间的对齐性质？)

Motivation

• satisfy eigenspace alignment directly
• initialize $W_p$ outside the basin of collapse.

Implementation

estimate $\hat{F}$ by moving average

$f=Wx$, ${\mathbb{E}}_B[\cdot ]$ is the expectation over a batch

Problem

most analysis is under symmetric $W_p$, but actually regular $W_p$ can also work.