梯度下降法

梯度下降法

随机梯度下降法

批梯度下降法

$$g_k=f_k^{\prime }(x)$$
$$x_{k+1}=x_k-lr\ast g_k$$

小批梯度下降法

优化算法

动量

$$g_k=f_k^{\prime }(x)$$
$$v_k=\alpha v_{k-1}-lr\ast g_k$$
$$x_{k+1}=x_k+v_k=x_k+\alpha v_k-lr\ast g_{k-1}$$

Nesterow 动量

$$g_k=f_k^{\prime }(x+\alpha v_k)$$
$$v_k=\alpha v_{k-1}-lr\ast g_k$$
$$x_{k+1}=x_k+v_k=x_k+\alpha v_k-lr\ast g_{k-1}$$

AdaGrad

$$g_k=f_k^{\prime }(x)$$
$$r_k=r_{k-1}+g\ast g$$
$$x_{k+1}=x_k-\frac{lr}{\delta +{\sqrt{r}}_k}\ast g_k$$

RMSProp

$$g_k=f_k^{\prime }(x)$$
$$r_k=\rho r_{k-1}+(1-\rho )g\ast g$$
$$x_{k+1}=x_k-\frac{lr}{\delta +{\sqrt{r}}_k}\ast g_k$$

Adam

$$g_k=f_k^{\prime }(x)$$
$$s_k={\rho }_1{s}_{k-1}+(1-{\rho }_1)g$$
$$r_k={\rho }_2{r}_{k-1}+(1-{\rho }_2)g\ast g$$
$$\hat{s}=\frac{s}{1-{\rho }_1^k}$$
$$\hat{r}=\frac{r}{1-{\rho }_2^k}$$
$$x_{k+1}=x_k-\frac{lr{\hat{s}}_k}{\delta +\sqrt{{\hat{r}}_k}}$$