凸优化问题
-
无约束优化问题
m i n f ( x ) min f(x) minf(x)
费马(Fermat)定理,直接 f ′ ( x ) = 0 f^{'}(x)=0 f′(x)=0求解 -
带等式约束的优化问题
m i n f ( x ) minf(x) minf(x)
s . t . h i ( x ) = 0 , i = 1 , 2 , ⋅ ⋅ ⋅ , n s.t. h_i(x)=0 , i=1,2,···,n s.t.hi(x)=0,i=1,2,⋅⋅⋅,n
拉格朗日(Largrange)乘子法,构造:
L ( x , α i ) = f ( x ) + ∑ i = 1 n α i h i ( x ) \displaystyle L(x, \alpha_i) = f(x) +\sum_{i=1}^{n} \alpha_ih_i(x) L(x,αi)=f(x)+i=1∑nαihi(x)
令 ∂ L ( x , α i ) ∂ x = 0 , ∂ L ( x , α i ) ∂ α i = 0 , i = 1 , 2 , ⋅ ⋅ ⋅ , n \frac{\partial L(x, \alpha_i)}{\partial x}=0 ,\frac{\partial L(x, \alpha_i)}{\partial \alpha_i}=0, i=1,2,···,n ∂x∂L(x,αi)=0,∂αi∂L(x,αi)=0,i=1,2,⋅⋅⋅,n -
带不等式约束的优化问题
m i n f ( x ) minf(x) minf(x)
s . t . h i ( x ) = 0 , i = 1 , 2 , ⋅ ⋅ ⋅ , n ; s.t. h_i(x)=0,i=1,2,···,n; s.t.hi(x)=0,i=1,2,⋅⋅⋅,n;
g j ( x ) ≤ 0 , j = 1 , 2 , ⋅ ⋅ ⋅ , m \quad g_j(x)\leq 0,j=1,2,···,m gj(x)≤0,j=1,2,⋅⋅⋅,m
构造Largrange函数:
L ( x , α i ) = f ( x ) + ∑ i = 1 n α i h i ( x ) + ∑ j = 1 m β j g j ( x ) \displaystyle L(x, \alpha_i) = f(x) + \sum_{i=1}^{n}\alpha_ih_i(x)+ \sum_{j=1}^{m}\beta_jg_j(x) L(x,αi)=f(x)+i=1∑nαihi(x)+j=1∑mβjgj(x)
此时需要使用一般化的拉格朗日乘子方法:KKT条件,函数的最优值必定满足下面条件:
(1)$ \frac{\partial L(x, \alpha_i)}{\partial x}=0 $;
(2) h i ( x ) = 0 , i = 1 , 2 , ⋅ ⋅ ⋅ , n h_i(x)=0,i=1,2,···,n hi(x)=0,i=1,2,⋅⋅⋅,n,另外 α i ≠ 0 \alpha_i \neq 0 αi̸=0;
(3) ∑ j = 1 m β j g j ( x ) = 0 ; β i ≥ 0 , j = 1 , 2 , ⋅ ⋅ ⋅ , m \displaystyle \sum_{j=1}^{m}\beta_jg_j(x)=0; \quad \beta_i≥0,j=1,2,···,m j=1∑mβjgj(x)=0;βi≥0,j=1,2,⋅⋅⋅,m
对于一般的任意问题而言,KKT条件是使一组解成为最优解的必要条件,当原问题是凸问题的时候,KKT条件也是充分条件。
前两个条件是针对极值和等式约束的,第三个条件来自不等式约束,函数的极值必须满足: g i ( x ) ≤ 0 g_i(x)\leq 0 gi(x)≤0,并且 α i ≥ 0 \alpha_i\geq 0 αi≥0,同时 ∑ j = 1 m β j g j ( x ) = 0 \displaystyle \sum_{j=1}^{m}\beta_jg_j(x)=0 j=1∑mβjgj(x)=0,这就要求要么某个不等式 g j ( x ) = 0 g_j(x)=0 gj(x)=0,要么其对应的 β j = 0 \beta_j=0 βj=0。
$\because \ \beta \geq0,g_k(x)\leq0 $
∴ β g ( x ) ≤ 0 \therefore \ \beta g(x)\leq0 ∴ βg(x)≤0
∴ m a x μ L ( x , β ) = f ( x ) \therefore \ \underset{\mu}{max} \ L(x,\beta)=f(x) ∴ μmax L(x,β)=f(x)
∴ m i n x f ( x ) = m i n x m a x β L ( x , β ) ( 1 ) \therefore \underset{x}{min}\ f(x)=\underset{x}{min} \ \underset{\beta}{max} \ L(x,\beta) \qquad(1) ∴xmin f(x)=xmin βmax L(x,β)(1)
交换:
m a x β m i n x L ( x , β ) \underset{\beta}{max} \ \underset{x}{min} \ L(x,\beta) βmax xmin L(x,β)
= m a x β [ m i n x f ( x ) + m i n x β g ( x ) ] =\underset{\beta}{max} \ [\underset{x}{min} \ f(x)+\underset{x}{min} \beta g(x)] =βmax [xmin f(x)+xminβg(x)]
= m a x β m i n x f ( x ) + m a x β m i n x β g ( x ) =\underset{\beta}{max} \underset{x}{min} \ f(x)+\underset{\beta}{max} \ \underset{x}{min} \beta g(x) =βmaxxmin f(x)+βmax xminβg(x)
此时, m a x β m i n x f ( x ) = m i n x f ( x ) \underset{\beta}{max} \underset{x}{min} f(x)=\underset{x}{min} f(x) βmaxxminf(x)=xminf(x),因为 f ( x ) f(x) f(x)与 β \beta β无关,而:
m a x β m i n x L ( x , β ) = m i n x f ( x ) , β g ( x ) = 0 , β = 0 ∣ ∣ g ( x ) = 0 \underset{\beta}{max} \ \underset{x}{min} \ L(x,\beta)= \underset{x}{min} \ f(x), \ \beta g(x)=0,\ \beta=0 \ ||\ g(x)=0 βmax xmin L(x,β)=xmin f(x), βg(x)=0, β=0 ∣∣ g(x)=0
因此:
m a x μ m i n x L ( x , μ ) = m i n x m a x μ L ( x , μ ) \underset{\mu}{max} \ \underset{x}{min} \ L(x,\mu)=\underset{x}{min} \ \underset{\mu}{max} \ L(x,\mu) μmax xmin L(x,μ)=xmin μmax L(x,μ)
s . t . μ g ( x ) = 0 s.t.\ \mu g(x)=0 s.t. μg(x)=0
μ ≥ 0 \qquad \mu \geq0 μ≥0
此时极小极大问题就转变成了极大极小问题,我们称,极大极小问题是原问题的对偶问题,条件就是KKT条件。
——2018.08
支持向量机笔记:
2017.1.6