Optimization Week 13: Newton method and barrier method

1 Newton method (Second order method)

1.1 Motivation

• Gradient methods is first order, which uses a linear approximation to iterate.
• Gradient method is not affine invariant. A linear or affine change will of variables will cahnge the convergence rate.
• Thus use the change to get the best convergence rate, which requires the Hessian to be identity, resulting in the second order method, Newton method.

1.2 Idea of Newton method

• Use local quadratic approximation
  $$g(x)=f(x_0)+\nabla f(x_0{)}^T(x-x_0)+(x-x_0{)}^T{\nabla }^{2}f(x_0)(x-x_0)$$ $$x_{+}=\underset{x}{\mathrm{argmin}}g(x)=x_0-[{\nabla }^{2}f(x_0{)}^T{]}^{-1}\nabla f(x_0)$$

1.3 Newton method

$$x_{+}=x_0-\eta [{\nabla }^{2}f(x_0){]}^{-1}\nabla f(x_0)$$

• Affine invariant
• Need to know Hessian ${\nabla }^{2}f$
• Converge fast
• Expensive each iteration

1.4 Step size

• $\eta =1$: pure newton, may not converge
• Backtracking line search (BLTS):
  $\alpha <1/2,\beta <1$
  $d=[{\nabla }^{2}f(x_0){]}^{-1}\nabla f(x_0)$
  while $f(x-\eta d)>f(x)-\alpha \eta \nabla f(x{)}^{T}d$
  $\eta =\eta \beta$
  Backtracking line search is a way to choose the step size by starting from a very optimistic step size, and then checking if we are too optimistic. And if we are, we decrease it by a fraction beta. And alpha is a parameter that decides whether we’re optimistic or not.

1.5 Convergence with BTLS

• Prerequisite 1: $mI\le {\nabla }^{2}f(x)\le MI$
• Prerequisite 2: ${\nabla }^{2}f(x)$ is L-lipschitz
• Two-phase convergence:
  • Damped phase ($||\nabla f(x)\ge \alpha ||$): $f(x_t)-f(x^{\ast })\le f(x_0)-f^{\ast }-\gamma t$
  • Pure phase ($||\nabla f(x)<\alpha ||$, BTLS selects $\eta =1$): $f(x_t)-f(x^{\ast })\le \frac{2m^3}{L^2}(\frac{1}{2}{)}^{2^{t-t_0+1}}$, or $\frac{L}{2m^2}||\nabla f(x_{+})||\le (\frac{L}{2m^2}||\nabla f(x)||{)}^2$.
• Steps to reach $\epsilon$ accuracy: $\frac{f(x_0)-f^{\ast }}{\gamma }+\log \log (\frac{{\epsilon }_0}{\epsilon })$, quadratic convergence
• Error $(\frac{1}{2}{)}^{2^t}$
• Gradient descent: $\log (\frac{1}{\epsilon })$, linear convergence.

1.6 Scale free Newton

1.6.1 Definition

• A one-dimensional convex function $f:\mathbb{R}\to \mathbb{R}$ is self-concordant if $|f^{\prime \prime \prime }|\le 2[f^{\prime \prime }(x){]}^{\frac{3}{2}}$.
• A n-dimensional convex function $f:{\mathbb{R}}^n\to \mathbb{R}$ is self-concordant if its every 1-d projection is self-concordant.

1.6.2 Convergence

Newton with BTLS$(\alpha ,\beta )$ for a self-concordant $f$ reaches $\epsilon$-

optimality in $c(\alpha ,\beta )(f(x_0)-f^{\ast })+\log \log (\frac{1}{\epsilon })$.

2 Quasi Newton methods

2.1 Motivation

• $x_{t+1}=x_t-\eta I\nabla f(x_t)$
• $x_{t+1}=x_t-\eta {\nabla }^{2}f(x_t{)}^{-1}\nabla f(x_t)$
• $x_{t+1}=x_t-\eta H_t\nabla f(x_t)$

2.2 Basic idea

$$x_{+}-x=s$$ Solve secant equation $$B_{+}s=\nabla f(x_{+})-\nabla f(x)$$ $$B_{+}s=y$$ for $B_{+}$ through cheap update of $B$ and keeping $B$ symmetric, positive semidefinite, then solve $$B_{+}p=-\nabla f(x)$$ for $p$ ,finally, approximate $$x_{+}=x_0-\eta [{\nabla }^{2}f(x_0){]}^{-1}\nabla f(x_0)$$ with $$x_{+}=x_0+\eta p$$

https://en.wikipedia.org/wiki/Quasi-Newton_method

Use Sherman–Morrison formula to:
Update of inverse

2.3 Convergence

• Super linear convergence when strong convexity + extra assumptions.

3 Barrier method (with constraints)

3.1 Basic idea

$$\begin{array}{rl}\underset{x}{\min }& f(x)\\ s.t.& h_i(x)\le C,i=1\dots m\\ & Ax=b\end{array}$$ Bring the constraints into the objective function using indicator function $I(u)=\infty$ if $u\ge 0$; $=0$ otherwise.
$$\begin{array}{rl}\underset{x}{\min }& f(x)+\sum _{i}I(h_i(x))\\ s.t.& Ax=b\end{array}$$ Then use a smooth function approx$=-\frac{1}{t}\log (-u)$ to approximate $I(u)$, $t\to \infty$, approx$\to I(u)$.

$$\begin{array}{rl}\underset{x}{\min }& tf(x)+[-\sum _{i=1}^m\log (-h_i(x))]\\ s.t.& Ax=b\end{array}$$

3.2 Barrier method

• Solve sequence of problems: $$\begin{array}{rl}\underset{x}{\min }& t^{k}f(x)+[-\sum _{i=1}^m\log (-h_i(x))]\\ s.t.& Ax=b\end{array}$$
• Start from initial $t^0$
• At each epoch $t^k$, find $x^{\ast }(t^k)$ using Newton method starting at $x^{\ast }(t^{k-1})$. Increase $t^{k+1}=\mu t^k$.

3.2.1 Central path

Solve the problem $$\begin{array}{rl}\underset{x}{\min }& t^{k}f(x)+[-\sum _{i=1}^m\log (-h_i(x))]\\ s.t.& Ax=b\end{array}$$ we get the optimal solution $x^{\ast }(t^k)$. $x^{\ast }(t^k)\to x^{\ast }$ when $k\to \infty$.

3.2.2 Choose $t$

• Initially, use small $t$ to avoid the bad conditioning, gradually increase $t$ when approaching $x^{\ast }$.