凸优化基础知识—对偶（Duality）

---

5.1 拉格朗日对偶函数（The Lagrange dual function）

5.1.1 拉格朗日（Lagrange）

An optimization problem in the standard form:
$$\begin{array}{l}\min f_0(x)\\ s.t.f_i(x)\le 0,i=1,...,m\\ h_i(x)=0,i=1,...,p\end{array}$$ with variable $x\in {\mathbb{R}}^n$. We assume its domain $\mathcal{D}={\bigcap }_{i=0}^m\mathrm{dom}{f}_i\cap {\bigcap }_{i=1}^p\mathrm{dom}{h}_i$ is nonempty, and denote the optimal value of the problem by $p^{\ast }$. We do not assume the problem is convex.

The basic idea in Lagrangian duality is to take the constraints into account by augmenting the objective function with a weighted sum of the constraint functions. We define the Lagrangian (function) $L$: ${\mathbb{R}}^n\times {\mathbb{R}}^m\times {\mathbb{R}}^p\to \mathbb{R}$ associated with the problem as
$$L(x,\lambda ,v)=f_0(x)+\sum _{i=1}^m{\lambda }_i{f}_i(x)+\sum _{i=1}^p{v}_i{h}_i(x)$$ with dom $L=D\times {\mathbb{R}}^m\times {\mathbb{R}}^p$. We refer to ${\lambda }_i$ as the Lagrange multiplier associated with the $i$th inequality constraint $f_i(x)\le 0$; similarly $v_i$ is the Lagrange multiplier associated with the $i$th inequality constraint $h_i(x)=0$. The vectors $\lambda$ and $\nu$ are called the dual variables or Lagrange multiplier vectors associated with the problem.

---

5.1.2 朗格朗日对偶函数（The Lagrange dual function）

Define Lagrange dual function $g$: ${\mathbb{R}}^m\times {\mathbb{R}}^p\to \mathbb{R}$ as the minimum value of the Lagrangian over $x$: for $\lambda \in {\mathbb{R}}^m$, $v\in {\mathbb{R}}^p$,
$$g(\lambda ,v)=\underset{x\in \mathcal{D}}{\inf }L(x,\lambda ,v)=\underset{x\in \mathcal{D}}{\inf }\left(f_0(x)+\sum _{i=1}^m{\lambda }_i{f}_i(x)+\sum _{i=1}^p{v}_i{h}_i(x)\right)$$
When the Lagrangian is unbounded below in $x$, the dual function takes on the value $-\infty$. Since the dual function is the pointwise infimum of a family of affine functions of $(\lambda ,\nu )$, it is concave, even when the problem is not convex.

---

5.1.3 最优值的下界（Lower bounds on optimal value）

The dual function yields lower bounds on the optimal value p ⋆ of the problem (5.1): For any $0⪯\lambda$ and any $\nu$ we have
$$g(\lambda ,v)=\underset{x\in \mathcal{D}}{\inf }L(x,\lambda ,v)\le p^{\ast },$$ since $$g(\lambda ,v)=\underset{x\in \mathcal{D}}{\inf }L(x,\lambda ,v)\le L(\tilde{x},\lambda ,v)\le f_0(\tilde{x})\le p^{\ast },$$ where $\tilde{x}$ is a feasible solution for the problem.

---

5.1.5 例子（Examples）

线性等式的最小二乘解（Least-squares solution of linear equations）

we consider the problem:
$$\begin{array}{l}\min x^{T}x\\ s.t.Ax=b\end{array}$$ where $A\in {\mathbb{R}}^{p\times n}$.
Then, we give the Lagrangian :
$$L(x,v)=x^{T}x+v^T(Ax-b),$$ with domain ${\mathbb{R}}^n\times {\mathbb{R}}^p.$
Since L(x,ν) is a convex quadratic function of $x$, we can find the minimizing x from the optimality condition
$${\nabla }_{x}L(x,v)=2x+A^{T}v=0,$$ which yields $x^{\ast }=-(\frac{1}{2}{A}^{T}v)$ . Thereforethe dual function is
$$g(v)=L\left(-(1/2)A^{T}v,v\right)=-(1/4)v^{T}A{A}^{T}v-b^{T}v$$, which is a concave quadratic function of $x$, with domain ${\mathbb{R}}^p$.

---

5.1.6 拉格朗日函数&共轭函数（The Lagrange dual function and conjugate functions）

the conjugate $f^{\ast }$ of a function $f$: ${\mathbb{R}}^n\to \mathbb{R}$ is given by
$$f^{\ast }(y)=\underset{x\in \mathbf{d}\mathbf{o}\mathbf{m}f}{\sup }(y^{T}x-f(x))$$

Given a problem:
$$\begin{array}{l}\min f(x)\\ s.t.x=0\end{array}$$ Then, we have Lagrangian $L(x,v)=f(x)+v^{T}x$, and dual function is $$g(v)=\underset{x}{\inf }(f(x)+v^{T}x)=-\underset{x}{\sup }((-v{)}^{T}x-f(x))=-f^{\ast }(-v)$$.
More generally, consider an optimization problem with linear inequality and equality constraints,
$$\begin{array}{l}\min f_0(x)\\ s.t.Ax⪯b\\ Cx=d.\end{array}$$
Using the conjugate of $f_0$, we can rewrite the dual function as follows,
$$\begin{array}{ll}g(\lambda ,v)& ={\inf }_x(f_0(x)+\lambda (Ax-b)+v^T(Cx-d))\\ & =-b^T\lambda -d^{T}v+{\inf }_x(f_0(x)+(A^T+C^{T}v{)}^{T}x)\\ & =-b^T\lambda -d^{T}v-f_0^{\ast }(-A^T\lambda -C^{T}v).\end{array}$$

---

5.2 朗格朗日对偶问题（The Lagrange dual problem）

The Lagrange dual problem of a Lagrange dual problem is primary problem.

For each pair $(\lambda ,v)$ with $\lambda >0$, the Lagrange dual function gives us a lower bound on the optimal value $p^{\ast }$ of the optimization problem. We can obtain from the Lagrange dual function by the optimization problem:
$$\begin{array}{l}\max g(\lambda ,v)\\ s.t.\lambda >0\end{array}$$ The above problem is called the Lagrange dual problem. The term dual feasible, to describe a pair $(\lambda ,v)$ with $\lambda >0$ and $g(\lambda ,v)>-\infty$, means, as the name implies, that $(\lambda ,v)$ is feasible for the dual problem. We refer to $({\lambda }^{\star },{\nu }^{\ast })$ as dual optimal or optimal Lagrange multipliers if they are optimal for the problem. The Lagrange dual problem is a convex optimization problem, since the objective to be maximized is concave and the constraint is convex.

5.2.1 明确双重约束（Making Dual constraints explicit）

The examples above show that it is not uncommon for the domain of the dual function, d o m   g = { ( λ , v )   ∣   g ( λ , v ) > − ∞ ) } \mathbf{dom} ~g = \{ (\lambda,v)~|~g(\lambda ,v)>-\infty ) \} dom g={(λ,v) ∣ g(λ,v)>−∞)}, to have dimension smaller than $m+p$, i.e., $\mathbf{d}\mathbf{o}\mathbf{m}g\in {\mathbb{R}}^{m+p}$.

A. 标准形式的朗格朗日对偶（Lagrange dual of standard from LP）

We found that the Lagrange dual function for the standard form LP
$$\begin{array}{l}\min c^{T}x\\ s.t.Ax=b\\ x⪰0\end{array}$$ is given by g ( λ , v ) = { − b T v ,   A T v − λ + c = 0 − ∞ , o t h e r w i s e g(\lambda,v) = \{ \begin{array}{rcl} -b^Tv, ~A^Tv-\lambda + c = 0 \\ -\infty, \qquad \quad otherwise \end{array} g(λ,v)={−bTv, ATv−λ+c=0−∞,otherwise​ Strictly speaking, the Lagrange dual problem of the standard form LP is to maximize this dual function g subject to $\lambda >0$, i.e., max ⁡   g ( λ , v ) = { − b T v ,   A T v − λ + c = 0 − ∞ , o t h e r w i s e s . t .     λ > 0 \begin{array}{ll} \max ~g(\lambda,v) = \{ \begin{array}{rcl} -b^Tv, ~A^Tv-\lambda + c = 0 \\ -\infty, \qquad \quad otherwise \end{array} \\ s.t. ~~~ \lambda > 0 \end{array} max g(λ,v)={−bTv, ATv−λ+c=0−∞,otherwise​s.t.   λ>0​
Here, $g$ is finite only when $A^{T}v-\lambda +c=0$
We can form an equivalent problem by making these equality constraints explicit: $$\begin{array}{l}\max -b^{T}v\\ s.t.A^{T}v-\lambda +c=0\\ \lambda ⪰0\end{array}$$
This problem, in turn, can be expressed as $$\begin{array}{l}\max -b^{T}v\\ s.t.A^{T}v+c⪰0\end{array}$$ which is an LP in inequality form.
Note that the first problem is the Lagrange dual of the standard form LP, which is equivalent to the two problems last.

B. 不等式形式线性规划的朗格朗日对偶（Lagrange Dual of Inequality Form LP）

In a similar way, we can find the Lagrange dual problem pf a linear program in inequality form
$$\begin{array}{ll}P0:& \min c^{T}x\\ & s.t.Ax⪯b.\end{array}$$ The Lagrangian is $$L(x,\lambda )=c^{T}x+{\lambda }^T(Ax-b)=-b^T\lambda +(A^T\lambda +c{)}^{T}x,$$ so the dual function is $$g(\lambda )=\underset{x}{\inf }L(x,\lambda )=-b^T\lambda +\underset{x}{\inf }(A^T\lambda +c{)}^{T}x.$$
So the dual function is
g ( λ ) = { − b T λ ,   A T λ + c = 0 − ∞ ,   o t h e r w i s e g(\lambda) = \{ \begin{array}{rcl} -b^T\lambda, ~A^T\lambda + c = 0 \\ -\infty, ~\quad otherwise \end{array} g(λ)={−bTλ, ATλ+c=0−∞, otherwise​
The dual variable $\lambda$ is dual feasible if $\lambda ⪰0$ and $A^T\lambda +c=0.$
The Langrange dual of the LP is to maximize $g$ over all $\lambda ⪰0$. Again we can reformulate the Lagrange dual by explicitly including the dual feasibility conditions as constraints, as in
$$\begin{array}{ll}P1:& \max b^T\lambda \\ & s.t.A^T\lambda +c=0,\\ & \lambda ⪰0,\end{array}$$ which is an LP in standard form.
Note that the Lagrange dual of the problem $P1$ is (equivalent to) the primal problem $P0$.

---

5.2.2 弱对偶（Weak Duality）

The optimal value of the Lagrange dual problem, which we denote $d^{\ast }$, is, by definition, the best lower bound on $p^{\ast }$ that can be obtained from the Lagrange dual function. In particular, we have the simple but important inequality, called as weak duality, $$d^{\ast }<p^{\ast },$$ which holds even if the original problem is not convex. The weak duality inequality holds even if $d^{\ast }$ and $p^{\ast }$ are infinite.
We refer to the difference $p^{\ast }-d^{\ast }$ as the optimal duality gap of the original problem, since it gives the gap between the optimal value of the primal problem and the best (i.e., greatest) lower bound on it that can be obtained from the Lagrange dual function.

---

5.2.3 强对偶&Slater的约束准则（Strong Duality & Slater’s Constraint Qualification）

If the equality $$d^{\ast }=p^{\ast }$$ holds, i.e., the optimal duality gap is zero, then we say that strong duality holds.

Strong duality does not, in general, hold. But if the primal problem is convex, i.e., of the form
$$\begin{array}{ll}P0:& \min f_0(x)\\ & s.t.f_i(x)\le b.i=1,...,m,\\ & Ax=b,\end{array}$$ with $f_0,...,f_m$ convex, we usually (but not always) have the strong duality.
Some conditions on the problem, under which strong holds, are called constraint qualifications. One simple constraint qualification is Salter’s condition: There exists an $x\in \mathbf{r}\mathbf{e}\mathbf{l}\mathbf{i}\mathbf{n}\mathbf{t}D$ such that $$f_i(x)<0,i=1,...,m,Ax=b.$$ Such a point is sometimes called strictly feasible, since the inequality constraints holds with strict inequalities. Slater’s theorem states that strong duality holds, if 1) Slater’s condition holds and 2) the problem is convex.
Slater’s condition can be refined when some of the inequality constraint functions $f_i$ are affine. If the first $k$ constraint functions $f_1,...,f_k$ are affine, then the strong duality holds provided the following condition holds: There exists an There exists an $x\in \mathbf{r}\mathbf{e}\mathbf{l}\mathbf{i}\mathbf{n}\mathbf{t}D$ such that $$f_i(x)\le 0,i=1,...,k,f_i(x)<0,i=k+1,...,m,Ax=b.$$

---

5.2.4 Examples

A. Lagrange dual of QCQP

We consider the QCQP
$$\begin{array}{ll}P0:& \min \frac{1}{2}{x}^T{P}_{0}x+q_0^{T}x+r_0\\ & s.t.\frac{1}{2}{x}^T{P}_{0}x+q_0^{T}x+r_0\le 0,i=1,...,m,\end{array}$$ with $P_0\in {\mathbf{S}}_{++}^n$ and $P_i\in {\mathbf{S}}_{+}^n$, $i=1,...,m$.
The Lagrangian is $$\begin{array}{ll}L(x,\lambda )& =\frac{1}{2}{x}^T{P}_{0}x+q_0^{T}x+r_0+{\sum }_{i=1}^m{\lambda }_i[\frac{1}{2}{x}^T{P}_{0}x+q_0^{T}x+r_0],i=1,...,m，\\ & =\frac{1}{2}{x}^{T}P(\lambda )x+q(\lambda {)}^{T}x+r(\lambda )\end{array}$$ where $P(\lambda )=P_0+{\sum }_{i=1}^m{\lambda }_i{P}_i,$ $q(\lambda )=q_0+{\sum }_{i=1}^m{\lambda }_i{q}_i,$ and $r(\lambda )=r_0+{\lambda }_i{r}_i.$
If $\lambda ⪰0$, however, we have $P(\lambda )\succ 0$ and $$g(\lambda )=\underset{x}{\inf }L(x,\lambda )=-\frac{1}{2}q(\lambda {)}^{T}P(\lambda {)}^{-1}q(\lambda )+r(\lambda ).$$ We can therefore express the dual problem as
$$\begin{array}{ll}P1:& \max g(\lambda )\\ & s.t.\lambda ⪰0\end{array}$$ The Slater condition says that strong duality between the primal problem $P0$ and the dual problem $P1$ holds if the quadratic inequality constraints are strictly feasible, i.e., there exists an $x$ with
$$(\frac{1}{2})x^T{P}_{i}x+q_i^{T}x+r_i<0,i=1,...,m.$$

B. A nonconvex quadratic problem with strong duality

On rare occasions, strong duality obtains for a nonconvex problem. As an important example, we consider the problem of minimizing a nonconvex quadratic function over the unit ball,
$$\begin{array}{ll}P0:& \max x^{T}Ax+2b^{T}x\\ & s.t.x^{T}x\le 1,\end{array}$$ where $A\in {\mathbf{S}}^n$ and $b\in {\mathbf{R}}^n$. When $A⋡0$, ths is not a convex problem. This problem is called the trust region problem.
The Lagrangian is $$L(x,\lambda )=x^{T}Ax+2b^{T}x+\lambda (x^{T}x-1)=x^T(A+\lambda \mathbf{I})x+2b^{T}x-\lambda ,$$
so the dual function is given by
g ( λ ) = { − b T ( A + λ I ) † b − λ , i f   A + λ I ⪰ 0 , b ∈ R ( A + λ I ) − ∞ , o t h e r w i s e , g(\lambda) = \{ \begin{array}{ll} -b^T(A+\lambda\mathbf{I})^{\dagger} b -\lambda, &\mathrm{if} ~ A+\lambda\mathbf{I} \succeq 0, b \in \mathcal{R}(A+\lambda\mathbf{I}) \\ -\infty, &otherwise, \end{array} g(λ)={−bT(A+λI)†b−λ,−∞,​if A+λI⪰0,b∈R(A+λI)otherwise,​
where $(A+\lambda \mathbf{I}{)}^{†}$ is the preudo-inverse of $(A+\lambda \mathbf{I})$. The Lagrange dual problem is thus
$$\begin{array}{ll}P1:& \max -b^T(A+\lambda \mathbf{I})†b-\lambda \\ & s.t.A+\lambda \mathbf{I}⪰0,b\in \mathcal{R}(A+\lambda \mathbf{I}),\end{array}$$ with variable $\lambda \in \mathbf{R}$.
The Lagrange dual problem is a convex optimization problem. In fact, it is readily solved since it can be expressed as
$$\begin{array}{l}\max -\frac{{\sum }_{i=1}^n(q_i^{T}b{)}^2}{({\lambda }_i+\lambda )}-\lambda \\ s.t.\lambda \ge -{\lambda }_{\min }(A),\end{array}$$ where ${\lambda }_i$ and $q_i$ are the eigenvalues and corresponding (orthonormal) eigenvectors of $A$, and we interpret $(q_i^{T}b{)}^2/0$ as $0$, if $q_i^{T}b=0$ and as $\infty$ otherwise.
Despite the original problem $P0$ is not convex, the strong duality still holds. In fact, a more general result holds: strong duality holds for any optimization problem with quadratic objective and one quadratic inequality constraint, provided Slater’s condition holds.

---

5.4 鞍点解释（Saddle-Point Interpretation）

5.4.1 Max-Min characterization of weak and strong duality

First note that
sup ⁡ λ ⪰ 0 L ( x , λ ) = sup ⁡ λ ⪰ 0 ( f 0 ( x ) + ∑ i = 1 m λ i f i ( x ) ) = { f 0 ( x ) ,   i f   f i ( x ) < 0 ,   i = 1 , . . . , m ∞ , o t h e r w i s e . \sup_{\lambda \succeq 0} L(x,\lambda) = \sup_{\lambda \succeq 0 } (f_0(x) + \sum_{i=1}^m \lambda_i f_i(x)) = \{ \begin{array}{ll} f_0(x), ~&\mathrm{if}~f_i(x)<0, ~ i =1,...,m \\ \infty, &otherwise. \end{array} λ⪰0sup​L(x,λ)=λ⪰0sup​(f0​(x)+i=1∑m​λi​fi​(x))={f0​(x), ∞,​if fi​(x)<0, i=1,...,motherwise.​
Suppose $x$ is not feasible, and $f_i(x)>0$ for some $i$. Then ${\sup }_{\lambda ⪰0}L(x,\lambda )=\infty ,$ as can be seen by choosing ${\lambda }_j=0,j\ne i,$ and ${\lambda }_i\to \infty$. On the other hand, if $f_i(x)<0$, $i=1,...,m$, then the optimal choice of $\lambda$ is $\lambda =0$ and ${\sup }_{\lambda ⪰0}L(x,\lambda )=f_0(x).$ This means that we can express the optimal value of the primal problem as $$p^{\ast }=\underset{x}{\inf }\underset{\lambda ⪰0}{\sup }L(x,\lambda ).$$
By the definition of the dual function, we also have optimal value of the dual problem $$d^{\ast }=\underset{\lambda ⪰0}{\sup }\underset{x}{\inf }L(x,\lambda ).$$
Thus, the weak duality can be expressed as the inequality $$d^{\ast }=\underset{\lambda ⪰0}{\sup }\underset{x}{\inf }L(x,\lambda )\le \underset{x}{\inf }\underset{\lambda ⪰0}{\sup }L(x,\lambda )=p^{\ast },$$ and strong duality as the equality $$\underset{\lambda ⪰0}{\sup }\underset{x}{\inf }L(x,\lambda )\le \underset{x}{\inf }\underset{\lambda ⪰0}{\sup }L(x,\lambda ).$$
Strong duality means that the order of the minimization over $x$ and the maximization over $\lambda ⪰0$ can be switched without affecting the result.
In fact, the inequality does not depend on any properties of $L:$ We have $$\underset{z\in \mathbf{Z}}{\sup }\underset{w\in \mathbf{W}}{\inf }f(w,z)\le \underset{w\in \mathbf{W}}{\inf }\underset{z\in \mathbf{Z}}{\sup }f(w,z)$$
for any $f:{\mathbf{R}}^n\times {\mathbf{R}}^m\to \mathbf{R}$ (and any $\mathbf{W}\subseteq {\mathbf{R}}^n$ and $\mathbf{Z}\subseteq {\mathbf{R}}^m$). This general inequality is called the max-min inequality. When equality holds, i.e., $$\underset{z\in \mathbf{Z}}{\sup }\underset{w\in \mathbf{W}}{\inf }f(w,z)=\underset{w\in \mathbf{W}}{\inf }\underset{z\in \mathbf{Z}}{\sup }f(w,z),$$ $f$ (and $\mathbf{W}$ and $\mathbf{Z}$) satisfy the strong max-min property or saddle-point property.

---

5.4.2 Saddle-Point Interpretation

We refer to a pair $\tilde{w}\in W,\tilde{z}\in Z$ as a saddle-point for $f$ (and $W$ and $Z$) if $$f(\tilde{w},z)\le f(\tilde{w},\tilde{z})\le f(w,\tilde{z})$$ for all $w\in W,z\in Z.$ In other words, $\tilde{w}$ minimizes $f(w,\tilde{z})$ (over $w\in W$) and $\tilde{z}$ maximizes $f(\tilde{w},z)$ (over $z\in Z$): $$f(\tilde{w},\tilde{z})=\underset{w\in W}{\inf }f(w,\tilde{z}),f(\tilde{w},\tilde{z})=\underset{z\in Z}{\sup }f(\tilde{w},z).$$ This implies that the strong max-min property holds, and that the common value if $f(\tilde{w},\tilde{z}).$
Returning to our discussion of Lagrange duality, we see that if $x^{\star }$ and ${\lambda }^{\star }$ are respectively primal and dual optimal points for a problem in which strong duality obtains, they form a saddle-point for the Lagrangian. The converse is also true: If $(x,\lambda )$ is a saddle-point of the Lagrangian, then x is primal optimal, λ is dual optimal, and the optimal duality gap is zero.

---

5.5 最优化条件（Optimality conditions）

5.5.1 Certificate of suboptimality and stopping criteria

If we can find a dual feasible $g(\lambda ,\nu ),$ we can establish a lower bound on the optimal value of the primal problem: $p^{\ast }\le g(\lambda ,\nu ).$ Thus， a dual feasible point (λ,ν) provides a proof or certificate that $g(\lambda ,\nu ).$

---

5.5.2 Complementary slackness

Let $x^{\star }$ be a primal optimal and $({\lambda }^{\star },{\nu }^{\star })$ be a dual optimal point. This means that
$$\begin{array}{ll}{f}_0(x^{\ast })& =g({\lambda }^{\ast },{\nu }^{\ast })\\ & ={\inf }_x(f_0(x)+{\sum }_{i=1}^m{\lambda }_i^{\ast }{f}_i(x)+{\sum }_{i=1}^m{\nu }_i^{\ast }{h}_i(x))\\ & \le f_0(x^{\ast })+{\sum }_{i=1}^m{\lambda }_i^{\ast }{f}_i(x^{\ast })+{\sum }_{i=1}^m{\nu }_i^{\ast }{h}_i(x^{\ast })\\ & \le f_0(x^{\ast })\end{array}$$

• The first line states that the optimal duality is zero.
• The second line is the definition of the dual function.
• The third line follows since the infimum of the Lagrangian over $x$ is less than or equal to its value at $x=x^{\ast }.$
• The last inequality follows from ${\lambda }_i^{\ast }\ge 0,$ $f_i(x^{\ast })\le 0,$ $i=1,...,m,$ and $h_i(x^{\ast })=0,i=1,...,p.$

We conclude that the two inequalities (3-4 lines) in this chain hold with equality.
The first conclusion: since the inequality in the third line is an equality, we conclude that $x^{\ast }\star$ minimizes $L(x,{\lambda }^{\ast },{\nu }^{\star })$ over $x$.
The second conclusion (Complementary Slackness): $$\sum _{i=1}^m{\lambda }_i^{\ast }{f}_i(x^{\ast })=0.$$
Since each term in this sum is nonpositive, we conclude that $${\lambda }_i^{\ast }{f}_i(x^{\ast })=0,i=1,...,m.$$ it holds for any primal optimal $x^{\star }$ and any dual optimal $({\lambda }^{\star },{\nu }^{\star })$ (when strong duality holds).
We can express the complementary slackness condition as $${\lambda }_i^{\ast }>0\to f_i(x^{\ast })=0,$$ or, equivalently $$f_i(x^{\ast })<0\to {\lambda }_i^{\ast }=0.$$ Roughly speaking, this means the ith optimal Lagrange multiplier is zero unless the $i$th constraint is active at the optimum.

---

5.5.3 KKT optimality conditions

We now assume that the functions $f_0,...,f_m,h_1,...,h_p$ are differentiable (and therefore have open domains), but we make no assumptions yet about convexity.

A. KKT conditions for nonconvex problems

As above, let $x^{\star }$ and $({\lambda }^{\star },{\nu }^{\star })$ be any primal and dual optimal points with zero duality gap. Since x ⋆ minimizes $L(x,\lambda \star ,\nu \star )$ over $x$, it follows that its gradient must vanish at $x^{\star }$ , i.e., $$\nabla f_0(x^{\ast })+\sum _{i=1}^m{\lambda }_i^{\ast }\nabla f_i(x^{\ast })+\sum _{i=1}^p{\nu }_i^{\ast }\nabla f_i(x^{\ast })=0.$$
Thus, we have
$$\begin{array}{rl}{f}_i\left(x^{\star }\right)& \le 0,i=1,\dots ,m\\ h_i\left(x^{\star }\right)& =0,i=1,\dots ,p\\ {\lambda }_i^{\star }& \ge 0,i=1,\dots ,m\\ {\lambda }_i^{\star }{f}_i\left(x^{\star }\right)& =0,i=1,\dots ,m\\ \nabla f_0\left(x^{\star }\right)+\sum _{i=1}^m{\lambda }_i^{\star }\nabla f_i\left(x^{\star }\right)+\sum _{i=1}^p{\nu }_i^{\star }\nabla h_i\left(x^{\star }\right)& =0,\end{array}$$ which are called the Karush-Kuhn-Tucker (KKT) conditions.
To summarize, for any optimization problem with differentiable objective and differentiable constraint functions for which strong duality obtains, any pair of primal and dual optimal points must satisfy the KKT conditions.

B. KKT conditions for convex problems

When the primal problem is convex, the KKT conditions are also sufficient for the points to be primal and dual optimal. In other words, if $f_i$ are convex and $h_i$ are affine, and $\tilde{x},\tilde{\lambda },\tilde{\nu }$ are any points that satisfy the KKT conditions
$$\begin{array}{rl}{f}_i\left(\tilde{x}\right)& \le 0,i=1,\dots ,m\\ h_i\left(\tilde{x}\right)& =0,i=1,\dots ,p\\ \widetilde{{\lambda }_i}& \ge 0,i=1,\dots ,m\\ \widetilde{{\lambda }_i}{f}_i\left(\tilde{x}\right)& =0,i=1,\dots ,m\\ \nabla f_0\left(\tilde{x}\right)+\sum _{i=1}^m\widetilde{{\lambda }_i}\nabla f_i\left(\tilde{x}\right)+\sum _{i=1}^p\widetilde{{\nu }_i}\nabla h_i\left(\tilde{x}\right)& =0,\end{array}$$ then $\tilde{x}$ and $(\tilde{\lambda },\tilde{\nu })$ are primal and dual optimal, with zero duality gap.

To see this, note that the first two conditions state that $\tilde{x}$ is primal feasible. Since $\widetilde{{\lambda }_i}\le 0,L(x,\tilde{\lambda },\tilde{\nu })$ is convex in $x$; the last KKT condition states that its gradient with respect to $x$ vanishes at $x=\tilde{x}$, so it follows that $\tilde{x}$ minimizes $L(x,\tilde{\lambda },\tilde{\nu })$ over $x$. From this we conclude that

$$\begin{array}{rl}g(\tilde{\lambda },\tilde{\nu })& =L(\tilde{x},\tilde{\lambda },\tilde{\nu })\\ & =f_0(\tilde{x})\\ & =f_0(\tilde{x})+\sum _{i=1}^m\widetilde{{\lambda }_i}{f}_i\left(\tilde{x}\right)+\sum _{i=1}^p\widetilde{{\nu }_i}{h}_i\left(\tilde{x}\right)\end{array}$$ where in the last line we use $h_i(\tilde{x})=0$ and $\widetilde{{\lambda }_i}{f}_i(\tilde{x})=0$. This shows that $\tilde{x}$ and ($\tilde{\lambda },\tilde{\nu }$) have zero duality gap, and therefore are primal and dual optimal.
In summary, for any convex optimization problem with differentiable objective and differentiable constraint functions, any points that satisfy the KKT conditions are primal and dual optimal, and have zero duality gap.
If a convex optimization problem with differentiable objective and differentiable constraint functions satisfies Slater’s condition, then the KKT conditions provide necessary and sufficient conditions for optimality: Slater’s condition implies that the optimal duality gap is zero and the dual optimum is attained, so $x$ is optimal if and only if there are $(\lambda ,\nu )$ that, together with $x$, satisfy the KKT conditions.
The KKT conditions play an important role in optimization. In a few special cases, it is possible to solve the KKT conditions analytically. More generally, many algorithms for convex optimization are conceived as, or can be interpreted as, methods for solving the KKT conditions.

Example 5.1

Equality constrained convex quadratic minimization. We consider the problem
$$\begin{array}{llc}P0:& \min & (\frac{1}{2})x^{T}Px+q^{T}x+r\\ & s.t.& Ax=b,\end{array}$$ where $P\in S_{+}^n.$
The KKT conditions for this problem is
$$\begin{array}{ll}\min & A{x}^{\ast }=b,\\ & P{x}^{\ast }+q+A^T\nu =0,\end{array}$$ which we can write as
$$\left[\begin{array}{cc}P& A^T\\ A& 0\end{array}\right]\left[\begin{array}{l}{x}^{\star }\\ {\nu }^{\star }\end{array}\right]=\left[\begin{array}{c}-q\\ b\end{array}\right].$$
Solving this set of $m+n$ equations in the $m+n$ variables $x^{\star },{\nu }^{\star }$ gives the optimal primal and dual variables for $P0$.

Example 5.2 Water-filling.

We consider the convex optimization problem
$$\begin{array}{llc}P0:& \min & -{\sum }_{i=1}^n\log ({\alpha }_i+x_i)\\ & s.t.& x⪰0,1^{T}x=1,\end{array}$$ where ${\alpha }_i>0$. This problem arises in information theory, in allocating power to a set of $n$ communication channels. The variable $x_i$ represents the transmitter power allocated to the ith channel, and $\log ({\alpha }_i+x_i)$ gives the capacity or communication rate of the channel, so the problem is to allocate a total power of one to the channels, in order to maximize the total communication rate.
Introducing Lagrange multipliers ${\lambda }^{\star }\in {\mathbb{R}}^n$ for the inequality constraints $x^{\star }⪰0$, and a multiplier ${\nu }^{\star }\in R$ for the equality constraint $1^{T}x=1$, we obtain the KKT conditions
$$\begin{array}{ll}{x}^{\ast }& ⪰0,\\ 1^{T}x& =1\\ {\lambda }^{\ast }& ⪰0,i=1,...,n\\ {\lambda }_i^{\ast }{x}_i^{\ast }& =0,\\ -\frac{1}{({\alpha }_i+x_i^{\ast })}-{\lambda }_i^{\ast }+{\nu }^{\ast }& =0,i=1,...,n.\end{array}$$ We can directly solve these equations to find $x^{\star }$, ${\lambda }^{\star }$, and ${\nu }^{\star }$. We start by noting that ${\lambda }^{\star }$ acts as a slack variable in the last equation, so it can be eliminated, leaving
$$\begin{array}{ll}{x}^{\ast }& ⪰0,\\ 1^{T}x& =1\\ x_i^{\ast }({\nu }^{\ast }-\frac{1}{({\alpha }_i+x_i^{\ast })})& =0,\\ {\nu }^{\ast }& \ge \frac{1}{{\alpha }_i+x_i^{\ast }},i=1,...,n.\end{array}$$

• If ${\nu }^{\star }<1/{\alpha }_i$ , this last condition can only hold if $x_i^{\star }>0$, which by the third condition implies that ${\nu }^{\star }=\frac{1}{{\alpha }_i+x_i^{\star }}$.
• Solving for $x_i^{\star }$, we conclude that $x_i^{\star }=\frac{1}{{\nu }^{\star }}-{\alpha }_i$ if ${\nu }^{\star }<\frac{1}{{\alpha }_i}$.
• If ${\nu }^{\star }\ge 1/{\alpha }_i$, then $x_i^{\star }>0$ is impossible, because it would imply ${\nu }^{\star }\ge \frac{1}{{\alpha }_i}>\frac{1}{{\alpha }_i+x_i^{\star }}$, which violates the complementary slackness condition.
• Therefore, $x_i^{\star }=0$ if ${\nu }^{\star }\ge 1/{\alpha }_i$.
  Thus we have
  x i ∗ = { 1 ν ∗ − α i , i f    ν ∗ < 1 α i   0 , i f    ν ∗ ≥ 1 α i x_i^* = \{\begin{array}{ll} \frac{1}{\nu^*} - \alpha_i, &\mathrm{if} ~~ \nu^* < \frac{1}{\alpha_i}\\ \quad~ 0,& \mathrm{if} ~~ \nu^* \geq \frac{1}{\alpha_i} \end{array} xi∗​={ν∗1​−αi​, 0,​if  ν∗<αi​1​if  ν∗≥αi​1​​ or, put more simply, x i ∗ = max ⁡ { 0 , 1 ν ∗ − α i } x_i^* =\max \{0,\frac{1}{\nu^*} - \alpha_i \} xi∗​=max{0,ν∗1​−αi​}.
  Substituting this expression for $x_i^{\star }$ into the condition $1^T{x}^{\star }=1$, we obtain
  ∑ i = 1 n max ⁡ { 0 , 1 ν ∗ − α i } = 1. \sum_{i=1}^n \max \{0,\frac{1}{\nu^*} - \alpha_i \} = 1. i=1∑n​max{0,ν∗1​−αi​}=1. The lefthand side is a piecewise-linear increasing function of $1/{\nu }^{\star }$ , with breakpoints at ${\alpha }_i$ , so the equation has a unique solution which is readily determined.

---

5.5.5 Solving the primal problem via the dual

if strong duality holds and a dual optimal solution $({\lambda }^{\star },{\nu }^{\star })$ exists, then any primal optimal point is also a minimizer of $L(x,{\lambda }^{\star },{\nu }^{\star })$. This fact sometimes allows us to compute a primal optimal solution from a dual optimal solution. More precisely, suppose we have strong duality and an optimal $({\lambda }^{\star },{\nu }^{\star })$ is known. Suppose that the minimizer of $L(x,{\lambda }^{\star },{\nu }^{\star })$, i.e., the solution of $$\min f_0(x)+\sum _{i=1}^m{\lambda }_i^{\star }{f}_i(x)+\sum _{i=1}^p{\nu }_i^{\star }{h}_i(x)$$ is unique.

Example 5.3 Entropy maximization.

We consider the entropy maximization problem
$$\begin{array}{ll}\min & f_0(x)={\sum }_{i=1}^n{x}_i\log x_i\\ \text{subject to}& Ax⪯b\\ & 1^{T}x=1\end{array}$$ with domain ${\mathbf{R}}_{++}^n,$ and its Lagrange dual problem
$$\begin{array}{ll}\text{ maximize }& -b^T\lambda -\nu -e^{-\nu -1}{\sum }_{i=1}^n{e}^{-a_i^T\lambda }\\ \text{ subject to }& \lambda ⪰0\end{array}$$ where $a_i$ are the columns of $A$. We assume that the weak form of Slater’s condition holds, i.e., there exists an $x\succ 0$ with $Ax⪯b$ and $1^{T}x=1$, so strong duality holds and an optimal solution $({\lambda }^{\star },{\nu }^{\star })$ exists.
Suppose we have solved the dual problem. The Lagrangian at (${\lambda }^{\star },{\nu }^{\star }$) is
$$L\left(x,{\lambda }^{\star },{\nu }^{\star }\right)=\sum _{i=1}^n{x}_i\log x_i+{\lambda }^{\star T}(Ax-b)+{\nu }^{\star }\left(1^{T}x-1\right)$$ which is strictly convex on $\mathcal{D}$ and bounded below, so it has a unique solution $x^{\star }$ , given by
$$x_i^{\ast }=1/\exp (a_i^T{\lambda }^{\ast }+{\nu }^{\ast }+1),i=1,...,n.$$
If $x^{\star }$ is primal feasible, it must be the optimal solution of the primal problem. If $x^{\star }$ is not primal feasible, then we can conclude that the primal optimum is not attained.

---

5.7 Examples (reformulations)

In this section, we show by example that simple equivalent reformulations of a problem can lead to very different dual problems. We consider the following types of reformulations:

• Introducing new variables and associated equality constraints.
• Replacing the objective with an increasing function of the original objective.
• Making explicit constraints implicit, i.e., incorporating them into the domain of the objective.

5.7.1 Introducing new variables and equality constraints

Consider an unconstrained problem of the form
$$P0:\min f_0(Ax+b).$$ Its Lagrange dual function is the constant $p^{\star }$ . So while we do have strong duality, i.e., $p^{\star }=d^{\star }$, the Lagrangian dual is neither useful nor interesting.

Now let us reformulate the problem as
$$\begin{array}{ll}P1:& \min f_0(Ax+b)\\ & s.t.Ax+b=y.\end{array}$$ Here we have introduced new variables y, as well as new equality constraints $Ax+b=y$. The problems $P0$ and $P1$ are clearly equivalent.
The Lagrangian of the reformulated problem is
$$L(x,y,\nu )=f_0(y)+{\nu }^T(Ax+b-y).$$ To find the dual function we minimize $L$ over $x$ and $y$. Minimizing over $x$, we find that $g(\nu )=-\infty$ unless $A^T\nu =0$, in which case we are left with
$$g(\nu )=b^T\nu +\underset{y}{\inf }(f_0(y)-{\nu }^{T}y)=b^T\nu -f_0^{\ast }(\nu ),$$ where $f_0^{\ast }$ is the conjugate of $f_0$. The dual problem of $P1$ can therefore be expressed as
$$\begin{array}{ll}P1:& \min g(\nu )=b^T-f_0^{\ast }(\nu )\\ & s.t.A^T\nu =0.\end{array}$$ Thus, the dual of the reformulated problem $P1$ is considerably more useful than the dual of the original problem $P0$.

Example 5.5 Unconstrained geometric program.

Consider the unconstrained geometric program
$$\min \log (\sum _{i=1}^m\exp (a_i^{T}x+b_i)).$$ We first reformulate it by introducing new variables and equality constraints:
$$\begin{array}{ll}P1:& \min f_0(y)=\log ({\sum }_{i=1}^m\exp (a_i^{T}x+b_i))\\ & s.t.Ax+b=y.\end{array}$$ where $a_i^T$ are the rows of $A$. The conjugate of the log-sum-exp function is
f 0 ∗ = { ∑ i = 1 m ν i log ⁡ ν i , i f   ν ⪰ 0 , 1 T ν = 1    ∞ o t h e r w i s e f_0^* = \{\begin{array}{ll} \sum_{i=1}^m \nu_i \log \nu_i, &\mathrm{if}~ \nu \succeq 0, \mathbf{1}^T\nu =1 \\ \qquad ~~ \infty &\mathrm{otherwise} \end{array} f0∗​={∑i=1m​νi​logνi​,  ∞​if ν⪰0,1Tν=1otherwise​ so the dual of the reformulated problem can be expressed
as $$\begin{array}{ll}\max & b^T\nu -{\sum }_{i=1}^m{\nu }_i\log {\nu }_i,\\ & 1^T\nu =1\\ & A^T\nu =0\\ & \nu ⪰0,\end{array}$$ which is an entropy maximization problem.

Example 5.6 Norm approximation problem.

We consider the unconstrained norm approximation problem
$$P0:\max \Vert Ax-b\Vert ,$$ where $\Vert \cdot \Vert$ is any norm. Here too the Lagrange dual function is constant, equal to the optimal value of $P0$, and therefore not useful.
Once again we reformulate the problem as
$$\begin{array}{ll}\min & \Vert y\Vert \\ & Ax-b=y.\end{array}$$
The Lagrange dual problem is,
$$\begin{array}{ll}\min & b^T\nu \\ & \Vert \nu {\Vert }_{\ast }\le 1\\ & A^T\nu =0,\end{array}$$ where we use the fact that the conjugate of a norm is the indicator function of the dual norm unit ball.
The idea of introducing new equality constraints can be applied to the constraint functions as well. Consider, for example, the problem
$$\begin{array}{ll}\min & b^T\nu \\ & \Vert \nu {\Vert }_{\ast }\le 1\\ & A^T\nu =0,\end{array}$$ where $A_i\in {\mathbf{R}}^{k_i\times n}$ and $f_i:{\mathbf{R}}^{k_i}\to \mathbf{R}$ are convex. We introduce a new variable $y_i\in {\mathbf{R}}^{k_i}$ , for $i=0,...,m$, and reformulate the problem as
$$\begin{array}{ll}\min & f_0(y_0)\\ & f_i(y_i)\le 0,i=1,...,m.\\ & A_{i}x+b_i=y_i,i=0,...,m.\end{array}$$
The Lagrangian for this problem is
$$L(x,y_0,...,\lambda ,{\nu }_o,...,{\nu }_m)=f_0(y_0)+\sum _{i=1}^m{\lambda }_i{f}_i(y_i)+\sum _{i=0}^m{\nu }_i^T(A_{i}x+b_i-y_i).$$
To find the dual function, we minimize over $x$ and $y_i$. The minimum over $x$ is $-\infty$ unless $$\sum _{i=0}^m{A}_i^T{\nu }_i=0,$$ in which case we have, for $\lambda \succ 0$,
$$\begin{array}{rl}& g\left(\lambda ,{\nu }_0,\dots ,{\nu }_m\right)\\ & =\sum _{i=0}^m{\nu }_i^T{b}_i+\underset{y_0,\dots ,y_m}{\inf }\left(f_0\left(y_0\right)+\sum _{i=1}^m{\lambda }_i{f}_i\left(y_i\right)-\sum _{i=0}^m{\nu }_i^T{y}_i\right)\\ & =\sum _{i=0}^m{\nu }_i^T{b}_i+\underset{y_0}{\inf }\left(f_0\left(y_0\right)-{\nu }_0^T{y}_0\right)+\sum _{i=1}^m{\lambda }_i\underset{y_i}{\inf }\left(f_i\left(y_i\right)-{\left({\nu }_i/{\lambda }_i\right)}^T{y}_i\right)\\ & =\sum _{i=0}^m{\nu }_i^T{b}_i-f_0^{\ast }\left({\nu }_0\right)-\sum _{i=1}^m{\lambda }_i{f}_i^{\ast }\left({\nu }_i/{\lambda }_i\right)\end{array}$$
The last expression involves the perspective of the conjugate function, and is therefore concave in the dual variables. Finally, we address the question of what happens when $\lambda \succ 0$, but some ${\lambda }_i$ are zero. If ${\lambda }_i=0$ and ${\nu }_i\ne 0$, then the dual function is $-\infty$. If ${\lambda }_i=0$ and ${\nu }_i=0$, however, the terms involving $y_i$, ${\nu }_i$, and ${\lambda }_i$ are all zero. Thus, the expression above for g is valid for all $\lambda \succ 0$, if we take ${\lambda }_i{f}_i^{\ast }({\nu }_i/{\lambda }_i)=0$ when ${\lambda }_i=0$ and ${\nu }_i=0$, and ${\lambda }_i{f}_i^{\ast }({\nu }_i/{\lambda }_i)=\infty$ when ${\lambda }_i=0$ and ${\nu }_i\ne 0$.
Therefore we can express the dual of the problem as
$$\begin{array}{ll}\min & {\sum }_{i=0}^m{\nu }_i^T{b}_i-f_0^{\ast }\left({\nu }_0\right)-{\sum }_{i=1}^m{\lambda }_i{f}_i^{\ast }\left({\nu }_i/{\lambda }_i\right)\\ & \lambda ⪰0\\ & {\sum }_{i=0}^m{A}_i^T{\nu }_i=0.\end{array}$$

5.7.2 Transforming the objective

If we replace the objective $f_0$ by an increasing function of $f_0$, the resulting problem is clearly equivalent. The dual of this equivalent problem, however, can be very different from the dual of the original problem.

Example 5.8

We consider again the minimum norm problem
$$\min \Vert Ax-b\Vert ,$$ where $\Vert \cdot \Vert$ is some norm. We reformulate this problem as
$$\begin{array}{rl}\min & \frac{1}{2}\Vert y{\Vert }^2\\ s.t.& Ax-b=y.\end{array}$$ Here we have introduced new variables, and replaced the objective by half its square. Evidently it is equivalent to the original problem.
The dual of the reformulated problem is
$$\begin{array}{rl}\min & -\frac{1}{2}\Vert y{\Vert }_{\ast }^2+b^T\nu \\ s.t.& A^T\nu =0.\end{array}$$ where we use the fact that the conjugate of $(1/2)\Vert \cdot {\Vert }^2$ is $(1/2)\Vert \cdot {\Vert }_{\ast }^2$.
Note that this dual problem is not the same as the dual problem (Example 5.6) derived earlier.

5.7.3 Implicit constraints

The next simple reformulation we study is to include some of the constraints in the objective function, by modifying the objective function to be infinite when the constraint is violated.

---