【Machine Learning】Optimization

本笔记基于清华大学《机器学习》的课程讲义梯度下降相关部分，基本为笔者在考试前一两天所作的Cheat Sheet。内容较多，并不详细，主要作为复习和记忆的资料。

Smoothness assumption

• Upper Bound for ${\nabla }^{2}f(x)$: $\Vert {\nabla }^{2}f(w)\Vert \le L$

  $f(w^{\prime })\le f(w)+⟨\nabla f(w),w^{\prime }-w⟩+\frac{L}{2}{\Vert w^{\prime }-w\Vert }^2$

  equivalent to $\Vert \nabla f(w)-\nabla f(w^{\prime })\Vert \le L\Vert w-w^{\prime }\Vert$

  • Proof:

    • $\Rightarrow$: $\Vert \nabla f(w)-\nabla f(w^{\prime })\Vert \le \Vert {\nabla }^{2}f(x)\Vert \Vert w-w^{\prime }\Vert \le L\Vert w-w^{\prime }\Vert$
    • $\Leftarrow$

  • 2-side: $\left|f(w^{\prime })-f(w)-⟨\nabla f(w),w^{\prime }-w⟩\right|\le \frac{L}{2}{\Vert w^{\prime }-w\Vert }^2$

  • contains both upper/lower bound

• When $w^{\prime }=w-\eta \nabla f(w)$, $\eta =\frac{1}{L}$ to make sure

  $f(w^{\prime })-f(w)=-\frac{1}{2\eta }{\Vert \nabla f(x)\Vert }^2<0$

Convex Function

• Lower Bound for ${\nabla }^{2}f(x)$

  $f(w^{\prime })\ge f(w)+\nabla f(w{)}^T(w^{\prime }-w)$

  • $\lambda \min {\nabla }^{2}f(w)\ge 0$

• Strong convex function: $f(w^{\prime })\ge f(w)+\nabla f(w{)}^T(w^{\prime }-w)+\frac{\mu }{2}{\Vert w-w^{\prime }\Vert }^2$

  • $\lambda \min {\nabla }^{2}f(w)\ge \mu \ge 0$

Convergence Analysis

• $f$ is $L$-smooth

• The sequence is $w_0,w_1,...,w_t$ and the optimized point is $w^{\ast }$,

• $w_i=w_{i-1}-\eta \nabla f(w_{i-1})$

• $\eta <\frac{2}{L}$

• Gradient Descent make progress

$$f(w^{\prime })-f(w)\le -\eta \left(1-\frac{L\eta }{2}\right){\Vert \nabla f(w)\Vert }^2$$

• Convex function: $1/t$ convergence rate

$$f(w_t)-f(w^{\ast })\le \frac{{\Vert w_0-w^{\ast }\Vert }^2}{2t\eta }$$

Stochastic Gradient Descent(SGD)

• $f$ is $L$-smooth and convex

• $w_{t+1}=w_t-\eta G_t,E[G_t]=\nabla f(w_t)$

$$Ef(\overline{w_t})-f(w^{\ast })\le \frac{{\Vert w_0-w^{\ast }\Vert }^2}{2t\eta }+\eta {\sigma }^2$$

• Convergence rate $1/\sqrt{t}$

SVRG

Proof: To read.

Mirror Descent

• After $k$ iterations, $f(\bar{x})-f(u)\le \frac{1}{k}{\sum }_{t=0}^{k-1}⟨\nabla f(x_t),x_t-u⟩$ (also calls regret)becomes smaller.

• $f$ is $\rho$-Lipschitz, that is $|\nabla f(x)|\le \rho$. After $T=O(\frac{{\rho }^2}{{ϵ}^2})$, $f(\bar{x})-f(x^{\ast })\le ϵ$. $1/\sqrt{t}$ convergence rate.

• Linear Coupling: $1/t^2$ convergence rate. $t\ge \Omega {\left(\frac{1}{ϵ}\right)}^{1/2}$