Prime Problem and Dual Problem

Prime Promblem and Dual Problem

recommend textbook:
Convex Optimization Stepthen Boyd
Nonlinear Programming

Prime Problem

$$\{\begin{array}{l}minimizef(\mathbf{\omega })\\ s.t.\{\begin{array}{l}{g}_i(\mathbf{\omega })\le 0(i=1\sim K)\\ h_i(\mathbf{\omega })=0(i=1\sim N)\end{array}\end{array}$$

Dual Problem

{ Θ ( α , β ) = min ⁡ a l l   ω { L ( ω , α , β ) } s . t .   α i ≥ 0 L ( ω , α , β ) = f ( ω ) + ∑ i = 1 K α i g i ( ω ) + ∑ i = 1 M β i h i ( ω ) L ( ω , α , β ) = f ( ω ) + α T g ( ω ) + β T h ( ω )   ( v e c t o r i z e ) \left\{\begin{array}{l} \Theta(\boldsymbol\alpha, \boldsymbol\beta) = \min\limits_{all \ ω}\{L(\boldsymbol\omega, \boldsymbol\alpha, \boldsymbol\beta)\}\\ s.t. \ \boldsymbol\alpha_i \geq0 \end{array}\right.\\ L(\boldsymbol\omega,\boldsymbol\alpha, \boldsymbol\beta) = f(\boldsymbol\omega)+\sum\limits_{i=1}^{K}\boldsymbol\alpha_ig_i(\boldsymbol\omega)+\sum\limits_{i=1}^{M}\boldsymbol\beta_ih_i(\boldsymbol\omega)\\ L(\boldsymbol\omega,\boldsymbol\alpha, \boldsymbol\beta) = f(\boldsymbol\omega)+\boldsymbol\alpha^Tg(\boldsymbol\omega)+\boldsymbol\beta^Th(\boldsymbol\omega) \ (vectorize) {Θ(α,β)=all ωmin​{L(ω,α,β)}s.t. αi​≥0​L(ω,α,β)=f(ω)+i=1∑K​αi​gi​(ω)+i=1∑M​βi​hi​(ω)L(ω,α,β)=f(ω)+αTg(ω)+βTh(ω) (vectorize)
Theorme: if ${\mathbf{\omega }}^{\ast }$ is the solution of the prime problem and ${\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast }$ is the solution of the dual problem. Then: $f({\mathbf{\omega }}^{\ast })\ge \Theta ({\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast })$
Prove:
∵ ${\mathbf{\omega }}^{\ast }$ satisfies prime problem and ${\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast }$ satisfy dual problem
$$L({\mathbf{\omega }}^{\ast },{\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast })=f({\mathbf{\omega }}^{\ast })+\sum _{i=1}^K\underset{\ge 0}{{\mathbf{\alpha }}_i^{\ast }}\underset{\le 0}{g({\mathbf{\omega }}^{\ast })}+\sum _{i=1}^N{\mathbf{\beta }}_i\underset{=0}{h(\mathbf{\omega })}$$
∴ $L({\mathbf{\omega }}^{\ast },{\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast })\le f({\mathbf{\omega }}^{\ast })$
it is easy to prove that:
$\Theta ({\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast })\le L({\mathbf{\omega }}^{\ast },{\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast })$
∴ $f({\mathbf{\omega }}^{\ast })\ge \Theta ({\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast })$

Define the distance between prime problem and dual problem Duality Gap $G=f({\mathbf{\omega }}^{\ast })-\Theta ({\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast })\ge 0$. In certain conditions, G = 0.

Strong Duality Therome(I won’t prove it here)
If f(ω) is a convex function and $g(\omega )=A\omega +b,h(\omega )=C\omega +d$, then G = 0.

Suppose we have already proved strong duality therome, then $f({\mathbf{\omega }}^{\ast })=\Theta ({\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast })$:
①${\omega }^{\ast }=\omega ,\Theta ({\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast })=L({\mathbf{\omega }}^{\ast },{\mathbf{\alpha }}^{\ast },{\mathbf{\beta }}^{\ast })$
②$\sum _{i=1}^K{\alpha }_i^{\ast }{g}_i({\omega }^{\ast })=0,\forall i=1\sim K,{\alpha }_i^{\ast }=0or{g}_i^{\ast }({\omega }^{\ast })=0$
② is very important, it is also called KKT condition