优化梯度下降算法

Optimization problem

speed up the training of your neural network

Normalizing inputs

1. subtract mean

$$\begin{gathered} \mu =\frac{1}{m}\sum _{i=1}^m{x}^{(i)} \\ x:=x-\mu \end{gathered}$$

2. normalize variance

$$\begin{gathered} {\sigma }^2=\frac{1}{m}\sum _{i=1}^m(x^{(i)}{)}^2 \\ x/=\sigma \end{gathered}$$

vanishing/exploding gradients

$$\begin{gathered} y=w^{[l]}{w}^{[l-1]}...w^{[2]}{w}^{[1]}x \\ {w}^{[l]}>I\to (w^{[l]}{)}^L\to \infty \\ {w}^{[l]}<I\to (w^{[l]}{)}^L\to 0 \end{gathered}$$

weight initialize

$$\begin{gathered} var(w)=\frac{1}{n^{(l-1)}} \\ {w}^{[l]}=np.random.randn(shape)\ast np.sqrt(\frac{1}{n^{(l-1)}}) \end{gathered}$$

gradient check

Numerical approximation

$$\begin{gathered} f(\theta )={\theta }^3 \\ {f}^{\prime }(\theta )=\frac{f(\theta +\epsilon )-f(\theta -\epsilon )}{2\epsilon } \end{gathered}$$

grad check

$$\begin{gathered} d{\theta }_{approx}[i]=\frac{J({\theta }_1,...{\theta }_i+\epsilon ...)-J({\theta }_1,...{\theta }_i-\epsilon ...)}{2\epsilon }=d\theta [i] \\ check:\frac{\Vert d{\theta }_{approx}-d\theta {\Vert }_2}{\Vert d{\theta }_{approx}{\Vert }_2+\Vert d\theta {\Vert }_2}<10^{-7} \end{gathered}$$

Optimize algorithm

mini-bach gradient

[ x ( 1 ) . . . x ( m ) ] → [ x { 1 } . . . x { m / u } ] ( a n      e p o c h : F o r w a r d      p r o p      o n      x { t } : z [ l ] = w [ l ] X { t } + b [ l ] A [ l ] = g [ l ] ( z [ l ] ) J { t } = 1 1000 ∑ i = 1 l L ( y ^ ( i ) , y ( i ) ) + λ 2 ∗ s i z e ∑ l ∥ w [ l ] ∥ F 2 B a c k w a r d      p r o p [x^{(1)}...x^{(m)}]\rightarrow [x^{\{1\}}...x^{\{m/u\}}]\\ (an\;\;epoch:Forward\;\;prop\;\;on\;\;x^{\{t\}}:\\ z^{[l]}=w^{[l]}X^{\{t\}}+b^{[l]}\\ A^{[l]}=g^{[l]}(z^{[l]})\\ J^{\{t\}}=\frac{1}{1000}\sum_{i=1}^l\mathcal{L}(\hat y^{(i)},y^{(i)})+\frac{\lambda}{2*size}\sum_l\Vert w^{[l]}\Vert_F^2\\ Backward\;\;prop [x(1)...x(m)]→[x{1}...x{m/u}](anepoch:Forwardproponx{t}:z[l]=w[l]X{t}+b[l]A[l]=g[l](z[l])J{t}=10001​i=1∑l​L(y^​(i),y(i))+2∗sizeλ​l∑​∥w[l]∥F2​Backwardprop

mini-batch size

size = m -> Batch gradient descent <- small train set (<2000)

size = 1 -> stochastic gradient descent

typical mini-batch size (62,128,256…)

exponential weighted averages

$$
v_\theta = 0\
\theta_t\rightarrow v_\theta:=\beta v_{\theta-1}+(1-\beta)\theta_\theta\

$$

Bias correction

$$\frac{1}{1-\beta }\to \frac{v_t}{1-{\beta }^t}$$

Momentum

$$\begin{gathered} V_{dw}=\beta V_{dw}+(1-\beta )dw \\ {V}_{db}=\beta V_{db}+(1-\beta )db \\ w:=w-\alpha V_{dw} \end{gathered}$$

RMSprop

$$\begin{gathered} S_{dw}={\beta }_2{S}_{dw}+(1-{\beta }_2)d{w}^2 \\ {S}_{db}={\beta }_2{S}_{db}+(1-{\beta }_2)d{b}^2 \\ w:=w-\alpha \frac{dw}{{\sqrt{S}}_{dw}+\epsilon } \end{gathered}$$

Adam algorithm

$$\begin{gathered} V_{dw}=0,S_{dw}=0 \\ {V}_{dw}={\beta }_1{V}_{dw}+(1-{\beta }_1)dw \\ {V}_{db}={\beta }_1{V}_{db}+(1-{\beta }_1)db \\ {S}_{dw}={\beta }_2{S}_{dw}+(1-{\beta }_2)d{w}^2 \\ {S}_{db}={\beta }_2{S}_{db}+(1-{\beta }_2)d{b}^2 \\ {V}_{dw}^{correct}=\frac{v_{dw}}{1-{\beta }_1^t} \\ {S}_{dw}^{correct}=\frac{s_{dw}}{1-{\beta }_2^t} \\ W:=W-\alpha \frac{V_{dw}^{correct}}{\sqrt{S_{dw}^{correct}}+\epsilon } \\ {\beta }_1:0.9,{\beta }_2:0.999 \end{gathered}$$

Learning rate decay

$$\begin{gathered} \alpha =\frac{1}{1+decayRate\ast epochNumber}{\alpha }_0 \\ \alpha =\frac{k}{\sqrt{epochNum}}{\alpha }_0 \end{gathered}$$

Local optima