Convex Optimization 读书笔记 (4)

Chapter5: Duality

5.1 The Lagrange dual function

5.1.1 The Lagrangian

Consider an optimization problem in the standard form
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$
We define the Lagrangian Duality $L:{\mathbf{R}}^n\times {\mathbf{R}}^m\times {\mathbf{R}}^p\to \mathbf{R}$ associated with the problem as
$$L(x,\lambda ,\nu )=f_0(x)+\sum _{i=1}^m{\lambda }_i{f}_i(x)+\sum _{i=1}^p{\nu }_i{h}_i(x)$$
with $\mathbf{d}\mathbf{o}\mathbf{m}L=\mathcal{D}\times \mathbf{R}\times \mathbf{R}$. We refer to ${\lambda }_i$ as the Lagrange multiplier associated with the $i$-th inequality constraint $f_i(x)\le 0$; similarly we refer to ${\nu }_i$ as the Lagrange multiplier associated with the $i$-th equality constraint $h_i(x)=0$. The vectors $\lambda$ and $\nu$ are called the dual variables or Lagrange multiplier vectors.

5.1.2 The Lagrange dual function

We define the Lagrange dual function $g:{\mathbf{R}}^m\times {\mathbf{R}}^p\to \mathbf{R}$ as the minimum value of the Lagrangian over $x:\mathrm{f}\mathrm{o}\mathrm{r}\lambda \in {\mathbf{R}}^m,\nu \in {\mathbf{R}}^p$,
$$g(\lambda ,\nu )=\underset{x\in \mathcal{D}}{\inf }L(x,\lambda ,\nu )$$

5.1.3 Lower bounds on optimal value

The dual function yields lower bounds on the optimal value $p^{\star }$ of the problem: For any $\lambda ⪰0$ and any $\nu$ we have
$$g(\lambda ,\nu )\le p^{\ast }$$

5.1.4 Linear approximation interpretation

5.1.5 Examples

5.1.6 The Lagrange dual function and conjugate functions

Consider an optimization problem with linear inequality and equality constraints,
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& Ax⪯b\\ & Cx=d\end{array}$$
Using the conjugate of $f_0$ we can write the dual function for the problem
$$\begin{array}{rl}g(\lambda ,\nu )& =\underset{x}{\inf }(f_0(x)+{\lambda }^T(Ax-b)+{\nu }^T(Cx-d))\\ & =-b^T\lambda -d^T\nu -f_0^{\ast }(-A^T\lambda -C^T\nu )\end{array}$$

5.2 The Lagrange dual problem

The optimization problem
$$\begin{array}{rl}\mathrm{m}\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& g(\lambda ,\nu )\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& \lambda ⪰b\end{array}$$
This problem is called the Lagrange dual problem. We refer to $({\lambda }^{\star },{\nu }^{\star })$ as dual optimal or optimal Lagrange multipliers if they are optimal for the problem. The Lagrange dual problem is a convex optimization problem whether the original one is convex or not.

5.2.1 Making dual constraints explicit

In many cases we can identify the affine hull of $\mathbf{d}\mathbf{o}\mathbf{m}g$, and describe it as a set of linear equality constraints.

5.2.2 Weak duality

The optimal value of the Lagrange dual problem, which we denote $d^{\star }$, is, by definition, the best lower bound on $p^{\star }$ that can be obtained from the Lagrange dual function:
$$d^{\ast }\le p^{\ast }$$
which holds even if the original problem is not convex. This property is called weak duality.

We refer to the difference $p^{\star }-d^{\star }$ as the optimal duality gap of the original problem.

5.2.3 Strong duality and Slater’s constraint qualification

If the equality
$$d^{\star }=p^{\star }$$
holds, then we say that strong duality holds.

One simple constraint qualification is Slater’s condition: There exists an $x\in \mathbf{r}\mathbf{e}\mathbf{l}\mathbf{i}\mathbf{n}\mathbf{t}\mathcal{D}$ such that
$$f_i(x)<0,i=1,...,m,Ax=b$$
and $f_0(x)$ is a convex function.

5.2.4 Examples

5.2.5 Mixed strategies for matrix games

5.3 Geometric interpretation

5.3.1 Weak and strong duality via set of values

Suppose G = { ( f 1 ( x ) , . . . , f m ( x ) , h 1 ( x ) , . . . , h p ( x ) , f 0 ( x ) ) ∈ R m × R p × R ∣ x ∈ D } \mathcal{G}=\{(f_1(x),...,f_m(x),h_1(x),...,h_p(x),f_0(x))\in \mathbf{R}^m\times\mathbf{R}^p\times\mathbf{R}\mid x\in \mathcal{D}\} G={(f1​(x),...,fm​(x),h1​(x),...,hp​(x),f0​(x))∈Rm×Rp×R∣x∈D}, then the dual function at $\lambda ,\nu$ is a supporting hyperplane to $\mathcal{G}$ :

5.3.2 Proof of strong duality under constraint qualification

5.3.3 Multicriterion interpretation

Take the scalarization method for the (unconstrained) multicriterion problem
$$\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}(\mathrm{w}.\mathrm{r}.\mathrm{t}{\mathbf{R}}_{+}^{m+1})F(x)=(f_1(x),...,f_m(x),f_0(x))$$
which is just the Lagrange duality for a problem without equality constraints
$${\tilde{\lambda }}^{T}F(x)=f_0(x)+\sum _{i=1}^m{\lambda }_i{f}_i(x)$$

5.4 Saddle-point interpretation

5.4.1 Max-min characterization of weak and strong duality

We can express the optimal value of the primal problem as
$$p^{\ast }=\underset{x}{\inf }\underset{\lambda ⪰0}{\sup }L(x,\lambda )$$
By the definition of the dual function, we also have
$$d^{\ast }=\underset{\lambda ⪰0}{\sup }\underset{x}{\inf }L(x,\lambda )$$
Thus, weak duality can be expressed as the inequality
$$\underset{\lambda ⪰0}{\sup }\underset{x}{\inf }L(x,\lambda )\le \underset{x}{\inf }\underset{\lambda ⪰0}{\sup }L(x,\lambda )$$
and strong duality as the equality
$$\underset{\lambda ⪰0}{\sup }\underset{x}{\inf }L(x,\lambda )=\underset{x}{\inf }\underset{\lambda ⪰0}{\sup }L(x,\lambda )$$

5.4.2 Saddle-point interpretation

Returning to our discussion of Lagrange duality, we see that if $x^{\star }$ and ${\lambda }^{\star }$ are primal and dual optimal points for a problem in which strong duality obtains, they form a saddle-point for the Lagrangian. The converse is also true: If $(x,\lambda )$ is a saddle-point of the Lagrangian, then x is primal optimal, $\lambda$ is dual optimal, and the optimal duality gap is zero.

5.4.3 Game interpretation

The optimal duality gap for the problem is exactly equal to the advantage afforded the player who goes second, i.e., the player who has the advantage of knowing his or her opponent’s choice before choosing. If strong duality holds, then there is no advantage to the players of knowing their opponent’s choice.

5.4.4 Price or tax interpretation

5.5 Optimality conditions

5.5.1 Certificate of suboptimality and stopping criteria

A dual feasible point $(\lambda ,\nu )$ provides a proof or certificate that $p^{\star }\ge g(\lambda ,\nu )$.

The stopping criterion
$$f_0(x^{(k)})-g({\lambda }^k,{\nu }^k)<{ϵ}_{\mathrm{a}\mathrm{b}\mathrm{s}}$$
guarantees that when the algorithm terminates, $x^{(k)}$ is ${ϵ}_{\mathrm{a}\mathrm{b}\mathrm{s}}$-suboptimal.

5.5.2 Complementary slackness

Let $x^{\star }$ be a primal optimal and $({\lambda }^{\star },{\nu }^{\star })$ be a dual optimal point. This condition is known as complementary slackness that ${\lambda }^{\ast }{f}_i(x^{\ast })=0$

5.5.3 KKT optimality conditions

KKT conditions for nonconvex problems

Let $x^{\star }$ be a primal optimal and $({\lambda }^{\star },{\nu }^{\star })$ be a dual optimal point. Since $x^{\star }$ minimizes $L(x,{\lambda }^{\star },{\nu }^{\star })$ over $x$, it follows that its gradient must vanish at $x^{\star }$,
$$\nabla f_0(x^{\star })+\sum _{i=1}^m{\lambda }_i^{\ast }{f}_i(x^{\star })+\sum _{i=1}^p{\nu }_i^{\ast }{h}_i(x^{\star })=0$$
then the Karush-Kuhn-Tucker (KKT) condition is
$$\begin{array}{rl}{f}_i(x^{\star })& \le 0,i=1,...,m\\ h_i(x^{\star })& =0,i=1,...,p\\ {\lambda }_i^{\ast }& \ge 0,i=1,...,m\\ {\lambda }_i^{\ast }{f}_i(x^{\star })& =0,i=1,...,m\\ \nabla f_0(x^{\star })+\sum _{i=1}^m{\lambda }_i^{\ast }{f}_i(x^{\star })+\sum _{i=1}^p{\nu }_i^{\ast }{h}_i(x^{\star })& =0\end{array}$$
To summarize, for any optimization problem with differentiable objective and constraint functions for which strong duality obtains, any pair of primal and dual optimal points must satisfy the KKT conditions.

KKT conditions for convex problems

If $f_i$ are convex and $h_i$ are affine, and $\tilde{x},\lambda ,\tilde{\nu }$ are any points that satisfy the KKT conditions, then $\tilde{x}$ and $(\lambda ,\tilde{\nu })$ are primal and dual optimal, with zero duality gap.

5.5.4 Mechanics interpretation of KKT conditions

5.5.5 Solving the primal problem via the dual

If strong duality holds and a dual optimal solution $({\lambda }^{\star },{\nu }^{\star })$ exists, then any primal optimal point is also a minimizer of $L(x,{\lambda }^{\star },{\nu }^{\star })$.

5.6 Perturbation and sensitivity analysis

5.6.1 The perturbed problem

Consider the following perturbed version of the original optimization problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le u_i,i=1,...m\\ & h_i(x)=v_i,i=1,...p\end{array}$$

5.6.2 A global inequality

Let $({\lambda }^{\star },{\nu }^{\star })$ be optimal for the dual of the unperturbed problem. Then for all $u$ and $\nu$ we have
$$p^{\ast }(u,v)\ge p^{\ast }(0,0)-{\lambda }^{\ast T}u-{\nu }^{\ast T}$$

5.6.3 Local sensitivity analysis

Provided strong duality holds, the optimal dual variables ${\lambda }^{\star },{\nu }^{\star }$ are related to the gradient of $p^{\star }$ at
$${\lambda }_i^{\ast }=-\frac{\partial p^{\ast }(0,0)}{\partial u_i},{\nu }_i^{\ast }=-\frac{\partial p^{\ast }(0,0)}{\partial v_i}$$
If ${\lambda }_i^{\star }$ is small, it means that the constraint can be loosened or tightened a bit without much effect on the optimal value; if ${\lambda }_i^{\star }$ is large, it means that if the constraint is loosened or tightened a bit, the effect on the optimal value will be great.

5.7 Examples

5.7.1 Introducing new variables and equality constraints

5.7.2 Transforming the objective

5.7.3 Implicit constraints

5.8 Theorems of alternatives

5.8.1 Weak alternatives via the dual function

We can apply Lagrange duality theory to the problem of determining feasibility of a system of inequalities and equalities
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& 0\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x)\le 0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$
This problem has optimal value
$$p^{\ast }=\{\begin{array}{rcl}& 0,& \mathrm{f}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{l}\mathrm{e}\\ & \infty ,& \mathrm{i}\mathrm{n}\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{s}\mathrm{i}\mathrm{b}\mathrm{l}\mathrm{e}\end{array}$$
Two systems of inequalities (and equalities) are called weak alternatives if at most one of the two is feasible.

5.8.2 Strong alternatives

When the original inequality system is convex and some type of constraint qualification holds, then the pairs of weak alternatives described above are strong alternatives, which means that exactly one of the two alternatives holds.

5.8.3 Examples

5.9 Generalized inequalities

Lagrange duality extends to a problem with generalized inequality constraints
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x){⪯}_{K_i}0,i=1,...m\\ & h_i(x)=0,i=1,...p\end{array}$$

5.9.1 The Lagrange dual

The Lagrange dual optimization problem is
$$\begin{array}{rl}\mathrm{m}\mathrm{a}\mathrm{x}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& g(\lambda ,\nu )\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& {\lambda }_i{⪰}_{K_i^{\ast }}b\end{array}$$
where $K_i^{\ast }$ denotes the dual cone of $K_i$.

5.9.2 Optimality conditions

Complementary slackness

We can conclude that
$${\lambda }_i^{\ast }{\succ }_{K_i}0⟹f_i(x^{\ast })=0,f_i(x^{\ast }){\prec }_{K_i}0⟹{\lambda }_i^{\ast }=0$$
However, in contrast to problems with scalar inequalities, it is possible to satisfy with ${\lambda }_i^{\star }\ne 0$ and $f_i(x^{\star })\ne 0$.

KKT conditions

Now we add the assumption that the functions $f_i,h_i$ are differentiable, and generalize the KKT conditions to problems with generalized inequalities:
$$\begin{array}{rl}{f}_i(x^{\star })& {⪯}_{K_i}0,i=1,...,m\\ h_i(x^{\star })& =0,i=1,...,p\\ {\lambda }_i^{\ast }& {⪰}_{K_i}0,i=1,...,m\\ {\lambda }_i^{\ast }{f}_i(x^{\star })& =0,i=1,...,m\\ \nabla f_0(x^{\star })+\sum _{i=1}^{m}D{f}_i(x^{\star }{)}^T{\lambda }_i^{\ast }+\sum _{i=1}^p{\nu }_i^{\ast }{h}_i(x^{\star })& =0\end{array}$$
where $D{f}_i(x^{\star })\in {\mathbf{R}}^{k_i\times n}$ is the derivative of $f_i$ evaluated at $x^{\star }$.

5.9.3 Perturbation and sensitivity analysis

We consider the associated perturbed version of the problem
$$\begin{array}{rl}\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}& f_0(x)\\ \mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{t}\mathrm{o}& f_i(x){⪯}_{K_i}{u}_i,i=1,...m\\ & h_i(x)=v_i,i=1,...p\end{array}$$

5.9.4 Theorems of alternatives

We can derive theorems of alternatives for systems of generalized inequalities and equalities
$$\begin{gathered} f_i(x){⪯}_{K_i}0,i=1,...m \\ {h}_i(x)=0,i=1,...p \end{gathered}$$
where $K_i\subseteq {\mathbf{R}}_i^k$ are proper cones. We will also consider systems with strict inequalities,
$$\begin{gathered} f_i(x){\prec }_{K_i}0,i=1,...m \\ {h}_i(x)=0,i=1,...p \end{gathered}$$