4. 前向/反向传播——Batch Normalization

参考资料

cs231n Course Materials: Backprop
Derivatives, Backpropagation, and Vectorization
cs231n Lecture 4:Neural Networks and Backpropagation
cs231n Assignment 2
笔记: Batch Normalization及其反向传播

4. Batch Normalization

前向传播

"""
	Forward pass for batch normalization.

    Input:
    - X: Data of shape (N, D)
    - gamma: Scale parameter of shape (D,)
    - beta: Shift paremeter of shape (D,)
    - bn_param: Dictionary with the following keys:
      - mode: 'train' or 'test'; required
      - eps: Constant for numeric stability
      - momentum: Constant for running mean / variance.
      - running_mean: Array of shape (D,) giving running mean of features
      - running_var Array of shape (D,) giving running variance of features

    Returns a tuple of:
    - Y: of shape (N, D)
    - cache: A tuple of values needed in the backward pass
"""

$${\mu }_j=\frac{1}{N}\sum _{i=1}^N{X}_{i,j}\text{\hspace{1em}(4.1)}$$
$${\sigma }_j^2=\frac{1}{N}\sum _{i=1}^N{\left(X_{i,j}-{\mu }_j\right)}^2\text{\hspace{1em}(4.2)}$$
$${\hat{X}}_{i,j}=\frac{X_{i,j}-{\mu }_j}{\sqrt{{\sigma }_j^2+ϵ}}\text{\hspace{1em}(4.3)}$$
在式(4.3)中，$ϵ$是一个非常小的正数，防止分母变成0。

在此基础上，引入可学习参数$\beta$、$\gamma$：
$$Y_{i,j}={\gamma }_j{\hat{X}}_{i,j}+{\beta }_j\text{\hspace{1em}(4.4)}$$
需要注意，式(4.1)和式(4.2)计算的均值和方差只在训练阶段使用，测试阶段使用的均值和方差可由训练阶段的均值和方差求滑动平均值得到。

反向传播

将上述计算过程进行更细的拆分，就可得到下面的计算图：

下面从后往前推导反向传播。
$$Y_{i,j}={\hat{X}}_{i,j}^{\gamma }+{\beta }_j\text{\hspace{1em}(4.5)}$$
由式(4.5)可得：
$$\begin{array}{rl}\frac{\partial L}{\partial {\beta }_j}& =\sum _i\frac{\partial L}{\partial Y_{i,j}}\frac{\partial Y_{i,j}}{\partial {\beta }_j}\\ & =\sum _i\frac{\partial L}{\partial Y_{i,j}}\cdot 1\\ & =\sum _i\frac{\partial L}{\partial Y_{i,j}}\end{array}\text{\hspace{1em}(4.6)}$$

$$\begin{array}{rl}\frac{\partial L}{\partial {\hat{X}}_{i,j}^{\gamma }}& =\frac{\partial L}{\partial Y_{i,j}}\frac{\partial Y_{i,j}}{\partial {\hat{X}}_{i,j}^{\gamma }}\\ & =\frac{\partial L}{\partial Y_{i,j}}\cdot 1\\ & =\frac{\partial L}{\partial Y_{i,j}}\end{array}\text{\hspace{1em}(4.7)}$$

---

$${\hat{X}}_{i,j}^{\gamma }={\gamma }_j{\hat{X}}_{i,j}\text{\hspace{1em}(4.8)}$$
由式(4.8)可得
$$\begin{array}{rl}\frac{\partial L}{\partial {\gamma }_j}& =\sum _i\frac{\partial L}{\partial {\hat{X}}_{i,j}^{\gamma }}\frac{\partial {\hat{X}}_{i,j}^{\gamma }}{\partial {\gamma }_j}\\ & =\sum _i\frac{\partial L}{\partial {\hat{X}}_{i,j}^{\gamma }}{\hat{X}}_{i,j}\end{array}\text{\hspace{1em}(4.9)}$$

$$\begin{array}{rl}\frac{\partial L}{\partial {\hat{X}}_{i,j}}& =\frac{\partial L}{\partial {\hat{X}}_{i,j}^{\gamma }}\frac{\partial {\hat{X}}_{i,j}^{\gamma }}{\partial {\hat{X}}_{i,j}}\\ & =\frac{\partial L}{\partial {\hat{X}}_{i,j}^{\gamma }}{\gamma }_j\end{array}\text{\hspace{1em}(4.10)}$$

---

$${\hat{X}}_{i,j}=(1/{\hat{\sigma }}_j)X_{i,j}^m\text{\hspace{1em}(4.11)}$$
由式(4.11)可得
$$\begin{array}{rl}\frac{\partial L}{\partial (1/{\hat{\sigma }}_j)}& =\sum _i\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{\partial {\hat{X}}_{i,j}}{\partial (1/{\hat{\sigma }}_j)}\\ & =\sum _i\frac{\partial L}{\partial {\hat{X}}_{i,j}}{X}_{i,j}^m\end{array}\text{\hspace{1em}(4.12)}$$

---

$$1/{\hat{\sigma }}_j=\frac{1}{{\hat{\sigma }}_j}\text{\hspace{1em}(4.13)}$$
由式(4.13)可得
$$\begin{array}{rl}\frac{\partial L}{\partial {\hat{\sigma }}_j}& =\frac{\partial L}{\partial (1/{\hat{\sigma }}_j)}\frac{\partial (1/{\hat{\sigma }}_j)}{\partial {\hat{\sigma }}_j}\\ & =-\frac{\partial L}{\partial (1/{\hat{\sigma }}_j)}\frac{1}{{\hat{\sigma }}_j^2}\\ & =-\frac{\partial L}{\partial (1/{\hat{\sigma }}_j)}\frac{1}{{\sigma }_j^2+ϵ}\end{array}\text{\hspace{1em}(4.14)}$$

---

$${\hat{\sigma }}_j=\sqrt{{\sigma }_j^2+ϵ}\text{\hspace{1em}(4.15)}$$
由式(4.15)可得
$$\begin{array}{rl}\frac{\partial L}{\partial {\sigma }_j^2}& =\frac{\partial L}{\partial {\hat{\sigma }}_j}\frac{\partial {\hat{\sigma }}_j}{\partial {\sigma }_j^2}\\ & =\frac{\partial L}{\partial {\hat{\sigma }}_j}\frac{1}{2\sqrt{{\sigma }_j^2+ϵ}}\end{array}\text{\hspace{1em}(4.16)}$$

---

$${\sigma }_j^2=\frac{1}{N}\sum _i{X}_{i,j}^2\text{\hspace{1em}(4.17)}$$
由式(4.17)可得
$$\begin{array}{rl}\frac{\partial L}{\partial X_{i,j}^2}& =\frac{\partial L}{\partial {\sigma }_j^2}\frac{\partial {\sigma }_j^2}{\partial X_{i,j}^2}\\ & =\frac{\partial L}{\partial {\sigma }_j^2}\frac{1}{N}\end{array}\text{\hspace{1em}(4.18)}$$

---

$$X_{i,j}^2={\left(X_{i,j}^m\right)}^2\text{\hspace{1em}(4.19)}$$
由式(4.11)和式(4.19)可得
$$\begin{array}{rl}\frac{\partial L}{\partial X_{i,j}^m}& =\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{\partial {\hat{X}}_{i,j}}{\partial X_{i,j}^m}+\frac{\partial L}{\partial X_{i,j}^2}\frac{\partial X_{i,j}^2}{\partial X_{i,j}^m}\\ & =\frac{\partial L}{\partial {\hat{X}}_{i,j}}(1/{\hat{\sigma }}_j)+\frac{\partial L}{\partial X_{i,j}^2}\cdot 2X_{i,j}^m\end{array}\text{\hspace{1em}(4.20)}$$

---

$$X_{i,j}^m=X_{i,j}-{\mu }_j\text{\hspace{1em}(4.21)}$$

$$\begin{array}{rl}\frac{\partial L}{\partial {\mu }_j}& =\sum _i\frac{\partial L}{\partial X_{i,j}^m}\frac{\partial X_{i,j}^m}{\partial {\mu }_j}\\ & =\sum _i\frac{\partial L}{\partial X_{i,j}^m}\cdot (-1)\\ & =-\sum _i\frac{\partial L}{\partial X_{i,j}^m}\end{array}\text{\hspace{1em}(4.22)}$$

---

$${\mu }_j=\frac{1}{N}\sum _i{X}_{i,j}\text{\hspace{1em}(4.23)}$$
由式(4.21)和式(4.23)得
$$\begin{array}{rl}\frac{\partial L}{\partial X_{i,j}}& =\frac{\partial L}{\partial X_{i,j}^m}\frac{\partial X_{i,j}^m}{\partial X_{i,j}}+\frac{\partial L}{\partial {\mu }_j}\frac{\partial {\mu }_j}{\partial X_{i,j}}\\ & =\frac{\partial L}{\partial X_{i,j}^m}\cdot 1+\frac{\partial L}{\partial {\mu }_j}\frac{1}{N}\end{array}\text{\hspace{1em}(4.24)}$$

以上，即为Batch Normalization的反向传播计算过程。

反向传播<简化版本>

实际上，可以将刚才的计算图进行简化(把一些节点合起来)，从而减少中间变量。然后这个图不知道还算不算计算图，，，其实就是按着最开始的公式硬算2333。
根据式(4.1-4)可得
$$\{\begin{array}{rl}{\mu }_j& =\frac{1}{N}\sum _i{X}_{i,j}\\ {\hat{\sigma }}_j& =\sqrt{\frac{1}{N}\sum _i{\left(X_{i,j}-{\mu }_j\right)}^2+ϵ}\\ {\hat{X}}_{i,j}& =\frac{X_{i,j}-{\mu }_j}{\widehat{{\sigma }_j}}\\ Y_{i,j}& ={\gamma }_j{\hat{X}}_{i,j}+{\beta }_j\end{array}\text{\hspace{1em}(4.25)}$$
$$\frac{\partial L}{\partial {\beta }_j}=\sum _i\frac{\partial L}{\partial Y_{i,j}}\frac{\partial Y_{i,j}}{\partial {\beta }_j}=\sum _i\frac{\partial L}{\partial Y_{i,j}}\text{\hspace{1em}(4.26)}$$
$$\frac{\partial L}{\partial {\gamma }_j}=\sum _i\frac{\partial L}{\partial Y_{i,j}}\frac{\partial Y_{i,j}}{\partial {\gamma }_j}=\sum _i\frac{\partial L}{\partial Y_{i,j}}{\hat{X}}_{i,j}\text{\hspace{1em}(4.27)}$$
$$\frac{\partial L}{\partial {\hat{X}}_{i,j}}=\frac{\partial L}{\partial Y_{i,j}}\frac{\partial Y_{i,j}}{\partial {\hat{X}}_{i,j}}=\frac{\partial L}{\partial Y_{i,j}}{\gamma }_j\text{\hspace{1em}(4.28)}$$

---

$$\begin{array}{rl}\frac{\partial L}{\partial {\hat{\sigma }}_j}& =\sum _{i=1}^N\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{\partial {\hat{X}}_{i,j}}{\partial {\hat{\sigma }}_j}\\ & =-\sum _{i=1}^N\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{X_{i,j}-{\mu }_j}{{\left(\widehat{{\sigma }_j}\right)}^2}\\ & =-\frac{1}{{\sigma }_j}\sum _{k=1}^N\frac{\partial L}{\partial {\hat{X}}_{k,j}}{\hat{X}}_{k,j}\end{array}\text{\hspace{1em}(4.29)}$$
$$\begin{array}{rl}\frac{\partial L}{\partial {\mu }_j}& =\frac{\partial L}{\partial {\hat{\sigma }}_j}\frac{\partial {\hat{\sigma }}_j}{\partial {\mu }_j}+\sum _{i=1}^N\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{\partial {\hat{X}}_{i,j}}{\partial {\mu }_j}\\ & =\frac{\partial L}{\partial {\hat{\sigma }}_j}\frac{\frac{1}{N}{\sum }_{i=1}^N-2\left(X_{i,j}-{\mu }_j\right)}{2\sqrt{\frac{1}{N}{\sum }_{i=1}^N{\left(X_{i,j}-{\mu }_j\right)}^2}}+\sum _{i=1}^N\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{\partial {\hat{X}}_{i,j}}{\partial {\mu }_j}\\ & =\frac{\partial L}{\partial {\hat{\sigma }}_j}\frac{\frac{1}{N}{\sum }_{i=1}^N\left({\mu }_j-X_{i,j}\right)}{\sqrt{\frac{1}{N}{\sum }_{i=1}^N{\left(X_{i,j}-{\mu }_j\right)}^2}}+\sum _{i=1}^N\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{\partial {\hat{X}}_{i,j}}{\partial {\mu }_j}\\ & =\frac{\partial L}{\partial {\hat{\sigma }}_j}\frac{0}{\sqrt{\frac{1}{N}{\sum }_{i=1}^N{\left(X_{i,j}-{\mu }_j\right)}^2}}+\sum _{i=1}^N\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{\partial {\hat{X}}_{i,j}}{\partial {\mu }_j}\\ & =\sum _{i=1}^N\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{\partial {\hat{X}}_{i,j}}{\partial {\mu }_j}\\ & =\sum _{i=1}^N\frac{\partial L}{\partial {\hat{X}}_{i,j}}\left(-\frac{1}{{\hat{\sigma }}_j}\right)\\ & =\sum _{k=1}^N\frac{\partial L}{\partial {\hat{X}}_{k,j}}\left(-\frac{1}{{\hat{\sigma }}_j}\right)\end{array}\text{\hspace{1em}(4.30)}$$

---

$$\begin{array}{rl}\frac{\partial L}{\partial X_{i,j}}& =\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{\partial {\hat{X}}_{i,j}}{\partial X_{i,j}}+\frac{\partial L}{\partial \widehat{{\sigma }_j}}\frac{\partial \widehat{{\sigma }_j}}{\partial X_{i,j}}+\frac{\partial L}{\partial {\mu }_j}\frac{\partial {\mu }_j}{\partial X_{i,j}}\\ & =\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{1}{{\hat{\sigma }}_j}+\frac{\partial L}{\partial \widehat{{\sigma }_j}}\frac{\frac{2}{N}\left(X_{i,j}-{\mu }_j\right)}{2\sqrt{\frac{1}{N}{\sum }_{i=1}^N{\left(X_{i,j}-{\mu }_j\right)}^2}}+\frac{\partial L}{\partial {\mu }_j}\frac{1}{N}\\ & =\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{1}{{\hat{\sigma }}_j}+\frac{\partial L}{\partial \widehat{{\sigma }_j}}\frac{\frac{2}{N}\left(X_{i,j}-{\mu }_j\right)}{2{\hat{\sigma }}_j}+\frac{\partial L}{\partial {\mu }_j}\frac{1}{N}\\ & =\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{1}{{\hat{\sigma }}_j}+\frac{\partial L}{\partial \widehat{{\sigma }_j}}\frac{1}{N}{\hat{X}}_{i,j}+\frac{\partial L}{\partial {\mu }_j}\frac{1}{N}\\ & =\frac{\partial L}{\partial {\hat{X}}_{i,j}}\frac{1}{{\hat{\sigma }}_j}-\left(\frac{1}{{\hat{\sigma }}_j}\sum _{k=1}^N\frac{\partial L}{\partial {\hat{X}}_{k,j}}{\hat{X}}_{k,j}\right)\frac{1}{N}{\hat{X}}_{i,j}-\sum _{k=1}^N\frac{\partial L}{\partial {\hat{X}}_{k,j}}\left(\frac{1}{{\hat{\sigma }}_j}\right)\frac{1}{N}\\ & =\frac{1}{N}\frac{1}{{\hat{\sigma }}_j}\left(N\frac{\partial L}{\partial {\hat{X}}_{i,j}}-{\hat{X}}_{i,j}\sum _{k=1}^N\frac{\partial L}{\partial {\hat{X}}_{k,j}}{\hat{X}}_{k,j}-\sum _{k=1}^N\frac{\partial L}{\partial {\hat{X}}_{k,j}}\right)\\ & =\frac{{\gamma }_j}{N{\hat{\sigma }}_j}\left(N\frac{\partial L}{\partial Y_{i,j}}-{\hat{X}}_{i,j}\sum _{k=1}^N\frac{\partial L}{\partial Y_{k,j}}{\hat{X}}_{k,j}-\sum _{k=1}^N\frac{\partial L}{\partial Y_{k,j}}\right)\end{array}\text{\hspace{1em}(4.31)}$$