Convex Optimization 读书笔记 (9)

Chapter10: Equality constrained minimization

10.1 Equality constrained minimization problems

Consider a optimization problem:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Ax=b\end{array}$$where $f:{\mathbf{R}}^n\to \mathbf{R}$ is convex and twice continuously differentiable, and $A\in {\mathbf{R}}^{p\times n}$ with $\mathbf{r}\mathbf{a}\mathbf{n}\mathbf{k}A=p<n$. Solving this is equal to solve the problem below:
$$A{x}^{\ast }=b,\nabla f(x^{\ast })+A^T{\nu }^{\ast }=0$$The first set of equations, $Ax=b$, are called the primal feasibility equations, which are linear. The second set of equations, $\nabla f(x^{\star })+A^T{\nu }^{\star }=0$, are called the dual feasibility equations, and are in general nonlinear.

10.1.1 Equality constrained convex quadratic minimization

Consider the equality constrained convex quadratic minimization problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f(x)=\frac{1}{2}{x}^{T}Px+q^{T}x+r\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Ax=b\end{array}$$where $P\in {\mathbf{S}}_{+}^n$ and $A\in {\mathbf{R}}^{p\times n}$. Here the optimality conditions are
$$A{x}^{\star }=b,P{x}^{\star }+q+A^T{\nu }^{\star }=0,$$which we can write as
$$\left[\begin{array}{cc}P& A^T\\ A& 0\end{array}\right]\left[\begin{array}{c}{x}^{\ast }\\ {\nu }^{\ast }\end{array}\right]=\left[\begin{array}{c}-q\\ b\end{array}\right]$$This set of $n+p$ linear equations in the $n+p$ variables $x^{\star },{\nu }^{\star }$ is called the KKT system for the equality constrained quadratic optimization problem.

10.1.2 Eliminating equality constraints

One general approach to solving the equality constrained problem is to eliminate the equality constraints, and then solve the resulting ${\mathbf{R}}^{n\times (n-p)}$ and vector $\tilde{x}\in {\mathbf{R}}^n$ that parametrize the (affine) feasible set:
{ x ∣ A x = b } = { F z + x ~ ∣ z ∈ R n − p } \{ x\mid Ax=b \}=\{ Fz+\tilde{x}\mid z\in \mathbf{R}^{n−p} \} {x∣Ax=b}={Fz+x~∣z∈Rn−p}Here $\tilde{x}$ can be chosen as any particular solution of $Ax=b$, and $F\in {\mathbf{R}}^{n\times (n-p)}$ is any matrix whose range is the nullspace of $A$. Then we minimize the problem:
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f(x)=(Fz+\tilde{x})\end{array}$$

10.1.3 Solving equality constrained problems via the dua

Another approach is to solve the dual, and then recover the optimal primal variable $x^{\star }$,
$$\mathrm{m}\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}-b^T\nu -f^{\ast }(-A^T\nu )$$

10.2 Newton’s method with equality constraints

The initial point must be feasible and we need to make sure that the Newton step $∆x_{\mathrm{n}\mathrm{t}}$ is a feasible direction: $A∆x_{\mathrm{n}\mathrm{t}}=0$.

10.2.1 The Newton step

We replace the objective with its second-order Taylor approximation near $x$, to form the problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& \hat{f}(x+v)=f(x)+\nabla f(x{)}^{T}v+\frac{1}{2}{v}^T{\nabla }^{2}f(x)v\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& A(x+v)=b\end{array}$$The Newton step of a constrained problem should satisfy
$$\left[\begin{array}{cc}{\nabla }^{2}f(x)& A^T\\ A& 0\end{array}\right]\left[\begin{array}{c}\Delta x_{\mathrm{n}\mathrm{t}}\\ w\end{array}\right]=\left[\begin{array}{c}-\nabla f(x)\\ 0\end{array}\right]$$

10.2.2 Newton’s method with equality constraints

The outline of Newton’s method with equality constraints is exactly the same as for unconstrained problems.

10.2.3 Newton’s method and elimination

10.2.4 Convergence analysis

In particular, the practical performance of New- ton’s method with equality constraints is exactly like the performance of Newton’s method for unconstrained problems. Once $x^{(k)}$ is near $x^{\star }$, convergence is extremely rapid, with a very high accuracy obtained in only a few iterations.

10.3 Infeasible start Newton method

10.3.1 Newton step at infeasible points

We substitute $x+∆x$ for $x^{\star }$ and $w$ for ${\nu }^{\star }$ in the optimality conditions, and use the first-order approximation
$$\nabla f(x+\Delta x)\approx \nabla f(x)+{\nabla }^{2}f(x)\Delta x$$for the gradient to obtain
$$A(x+∆x)=b,\nabla f(x)+{\nabla }^{2}f(x)∆x+A^{T}w=0.$$This is a set of linear equations for $∆x$ and $w$,
$$\left[\begin{array}{cc}{\nabla }^{2}f(x)& A^T\\ A& 0\end{array}\right]\left[\begin{array}{c}\Delta x_{\mathrm{n}\mathrm{t}}\\ w\end{array}\right]=\left[\begin{array}{c}-\nabla f(x)\\ Ax-b\end{array}\right]$$

10.3.2 Infeasible start Newton method

10.3.3 Convergence analysis

We show that once the norm of the residual is small enough, the algorithm takes full steps (which implies that feasibility is achieved), and convergence is subsequently quadratic. We also show that the norm of the residual is reduced by at least a fixed amount in each iteration before the region of quadratic convergence is reached.

10.3.4 Convex-concave games

We can use the infeasible start Newton method to compute a solution of a convex-concave game with twice differentiable payoff function. We define the residual as
$$r(u,v)=\nabla f(u,v)=\left[\begin{array}{c}{\nabla }_{u}f(u,v)\\ {\nabla }_{v}f(u,v)\end{array}\right]$$

10.3.5 Examples

10.4 Implementation

10.4.1 Elimination

To implement the elimination method, we have to calculate a full rank matrix $F$ and an $\tilde{x}$ such that
{ x ∣ A x = b } = { F z + x ~ ∣ z ∈ R n − p } \{ x\mid Ax=b \}=\{ Fz +\tilde{x}\mid z\in\mathbf{R}^{n-p} \} {x∣Ax=b}={Fz+x~∣z∈Rn−p}

10.4.2 Solving KKT systems

10.4.3 Examples