Convex Optimization 读书笔记 (10)

Chapter11: Interior-point methods

11.1 Inequality constrained minimization problems

In this chapter we discuss interior-point methods for solving convex optimization problems that include inequality constraints,
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 0,i=1,...m\\ & Ax=b\end{array}$$where $f_0,...,f_m:{\mathbf{R}}^n\to \mathbf{R}$ are convex and twice continuously differentiable, and $A\in {\mathbf{R}}^{p\times n}$ with $\mathbf{r}\mathbf{a}\mathbf{n}\mathbf{k}A=p<n$.

11.2 Logarithmic barrier function and central path

Our first step is to rewrite the problem, making the inequality constraints implicit in the objective:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)+\sum _{i=1}^m{I}_{-}(f_i(x))\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Ax=b\end{array}$$where $I_{-}$ is the indicator function:
$$I_{-}(u)=\{\begin{array}{cc}0& u<0\\ \infty & u\ge 0\end{array}$$

11.2.1 Logarithmic barrier

The basic idea of the barrier method is to approximate the indicator function $I_{-}$ by the function
$${\hat{I}}_{-}(u)=-\frac{1}{t}\log (-u),\mathbf{d}\mathbf{o}\mathbf{m}{\hat{I}}_{-}=-{\mathbf{R}}_{++}$$The objective here is convex, since $-\frac{1}{t}\log (-u)$ is convex and increasing in $u$, and differentiable.

The function
$$\varphi (x)=-\sum _{i=1}^m\log (-f_i(x))$$
with d o m   ϕ = { x ∈ R n ∣ f i ( x ) < 0 , i = 1 , . . . , m } \mathbf{dom} \ \phi = \{x ∈ \mathbf{R}^n | f_i(x) < 0, i = 1,...,m\} dom ϕ={x∈Rn∣fi​(x)<0,i=1,...,m}, is called the logarithmic barrier or log barrier for the problem.

11.2.2 Central path

We now consider in more detail the minimization problem. It will simplify notation later on if we multiply the objective by $t$, and consider the equivalent problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& t{f}_0(x)+\varphi (x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Ax=b\end{array}$$For $t>0$ we define $x^{\star }(t)$ as the solution. The central path associated with problem is defined as the set of points $x^{\star }(t),t>0$, which we call the central points.

11.3 The barrier method

We have seen that the point $x^{\star }(t)$ is $m/t$-suboptimal, and that a certificate of this accuracy is provided by the dual feasible pair ${\lambda }^{\star }(t)$, ${\nu }^{\star }(t)$. This suggests a very straightforward method for solving the original problem with a guaranteed specified accuracy $ϵ$: We simply take $t=\frac{m}{ϵ}$ and solve the equality constrained
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& \frac{m}{ϵ}{f}_0(x)+\varphi (x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Ax=b\end{array}$$

11.3.1 The barrier method

We compute $x^{\star }(t)$ for a sequence of increasing values of $t$, until $t\ge \frac{m}{ϵ}$, which guarantees that we have an $ϵ$-suboptimal solution of the original problem.

11.3.2 Examples

11.3.3 Convergence analysis

11.3.4 Newton step for modified KKT equations

Consider the modified KKT equations
$$\begin{array}{rl}\nabla f_0(x)+\sum _{i=1}^m{\lambda }_i{f}_i(x)+A^T\nu & =0\\ -{\lambda }_i{f}_i(x)& =\frac{1}{t},i=1,...,m\\ Ax& =b\end{array}$$Use ${\lambda }_i=\frac{1}{t{f}_i(x)}$, then we get
$$\nabla f_0(x)+\sum _{i=1}^m\frac{1}{-t{f}_i(x)}\nabla f_i(x)+A^T\nu =0,Ax=b$$For $v$ small, we use the Taylor approximation to approximate $\nabla f_0(x)+{\sum }_{i=1}^m{\lambda }_i{f}_i(x)$, then we get
$$Hv+A^T\nu =-g,Av=0,$$where
$$\begin{array}{rl}H& ={\nabla }^2{f}_0(x)+\sum _{i=1}^m\frac{1}{-t{f}_i(x)}{\nabla }^2{f}_i(x)+\sum _{i=1}^m\frac{1}{t{f}_i^2(x)}\nabla f_i(x)\nabla f_i(x{)}^T\\ g& =\nabla f_0(x)+\sum _{i=1}^m\frac{1}{-t{f}_i(x)}\nabla f_i(x)\end{array}$$Observe that
$$H={\nabla }^2{f}_0(x)+\frac{1}{t}{\nabla }^2\varphi (x),g=\nabla f_0(x)+\frac{1}{t}\nabla \varphi (x),$$We get that
$$v=∆x_{\mathrm{n}\mathrm{t}},\nu =\frac{1}{t}{\nu }_{\mathrm{n}\mathrm{t}}$$

11.4 Feasibility and phase I methods

When such a point is not known, the barrier method is preceded by a preliminary stage, called phase I, in which a strictly feasible point is computed (or the constraints are found to be infeasible). The strictly feasible point found during phase I is then used as the starting point for the barrier method, which is called the phase II stage.

11.4.1 Basic phase I method

Our goal is to find a strictly feasible solution of these inequalities and equalities, or determine that none exists. To do this we form the following optimization problem:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& s\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le s,i=1,...m\\ & Ax=b\end{array}$$

11.4.2 Phase I via infeasible start Newton method

We can also carry out the phase I stage using an infeasible start Newton method, applied to a modified version of the original problem. First express the problem in such form:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le s,i=1,...m\\ & Ax=b,s=0\end{array}$$
Provided the problem is strictly feasible, the infeasible start Newton method will eventually take an undamped step, and thereafter we will have $s=0$, i.e., $x$ strictly feasible.

11.4.3 Examples