6 函数估计的方法
6.1 ϵ \epsilon ϵ 不敏感损失函数
1964年 Huber 提出,如果我们只知道描述噪声的密度是一个对称函数,那么在最坏的噪声模型下最好的逼近(最小最大策略)为采用损失函数 L ( y , f ( x , α ) ) = ∣ y − f ( x , α ) ∣ L(y, f(x, \alpha)) = |y -f(x, \alpha)| L(y,f(x,α))=∣y−f(x,α)∣, 在这个损失函数下最小化经验风险的方法称为最小模方法,属于鲁棒回归(robust regression)方法。
Huber还考虑了正态噪声与某一对称噪声混合的情况,此时最优解在使用如下损失函数时得到
L ( ∣ y − f ( x , α ) ∣ ) = { c ∣ y − f ( x , α ) ∣ − c 2 2 i f ∣ y − f ( x , α ) ∣ > c 1 2 ∣ y − f ( x , α ) ∣ 2 i f ∣ y − f ( x , α ) ∣ ≤ c L(|y-f(x,\alpha)|) = \begin{cases} c|y-f(x,\alpha)| - \dfrac{c^2}{2} &if \ \ |y-f(x,\alpha)| > c \\ \dfrac{1}{2}|y-f(x,\alpha)|^2 &if \ \ |y-f(x,\alpha)| \le c \end{cases} L(∣y−f(x,α)∣)=⎩⎪⎨⎪⎧c∣y−f(x,α)∣−2c221∣y−f(x,α)∣2if ∣y−f(x,α)∣>cif ∣y−f(x,α)∣≤c
为了对实值函数(近似上面的损失函数)构造SVM,我们引入 ϵ \epsilon ϵ不敏感损失函数
∣ y − f ( x , α ) ∣ ϵ = ( ∣ y − f ( x , α ) ∣ − ϵ ) + |y-f(x,\alpha)|_\epsilon = (|y-f(x,\alpha)| - \epsilon)_+ ∣y−f(x,α)∣ϵ=(∣y−f(x,α)∣−ϵ)+
6.2 用于回归函数估计的SVM
对固定结构的解
设存在结构
S n = { ∣ y − ( w ⋅ x + b ) ∣ ϵ : w ⋅ w ≤ c n } S_n= \{|y-(w \cdot x + b)|_\epsilon:w \cdot w \le c_n\} Sn={
∣y−(w⋅x+b)∣ϵ:w⋅w≤cn}
经验风险
R e m p ( w , b ) = 1 l ∑ i = 1 l ∣ y − ( w ⋅ x + b ) ∣ ϵ R_{emp}(w, b) = \dfrac{1}{l} \sum\limits_{i=1}^l |y-(w \cdot x + b)|_\epsilon Remp(w,b)=l1i=1∑l∣y−(w⋅x+b)∣ϵ
设
∣ y i − ( w ⋅ x i + b ) ∣ ϵ = ξ i 0 → ∣ y i − ( w ⋅ x i + b ) ∣ ≤ ϵ + ξ i 0 → y i − ( w ⋅ x i + b ) ≤ ϵ + ξ i ∗ , ( w ⋅ x i + b ) − y i ≤ ϵ + ξ i → R e m p ( w , b ) = ∑ i = 1 l ξ i + ∑ i = 1 l ξ i ∗ |y_i-(w \cdot x_i + b)|_\epsilon = \xi_{i0} \to |y_i-(w \cdot x_i + b)| \le \epsilon + \xi_{i0} \\ \to y_i-(w \cdot x_i + b) \le \epsilon + \xi^*_i, \ \ (w \cdot x_i + b) - y_i \le \epsilon + \xi_i\\ \to R_{emp}(w, b) = \sum\limits_{i=1}^l \xi_i + \sum\limits_{i=1}^l \xi^*_i ∣yi−(w⋅xi+b)∣ϵ=ξi0→∣yi−(w⋅xi+b)∣≤ϵ+ξi0→yi−(w⋅xi+b)≤ϵ+ξi∗, (w⋅xi+b)−yi≤ϵ+ξi→Remp(w,b)=i=1∑lξi+i=1∑lξi∗
上式省略了常数系数 1 / l 1/l 1/l。显然上式最小化后会有 ξ i ξ i ∗ = 0 \xi_i \xi^*_i = 0 ξiξi∗=0 故 ξ i + ξ i ∗ = ξ i 0 \xi_i + \xi^*_i = \xi_{i0} ξi+ξi∗=ξi0。故有约束条件
y i − ( w ⋅ x i + b ) ≤ ϵ + ξ i ∗ , ( w ⋅ x i + b ) − y i ≤ ϵ + ξ i ξ i ≥ 0 , ξ i ∗ ≥ 0 , w ⋅ w ≤ c n y_i-(w \cdot x_i + b) \le \epsilon + \xi^*_i,\ \ \ (w \cdot x_i + b) - y_i \le \epsilon + \xi_i\\ \xi_i \ge 0,\ \ \ \xi^*_i \ge 0,\ \ \ w \cdot w \le c_n yi−(w⋅xi+b)≤ϵ+ξi∗, (w⋅xi+b)−yi≤ϵ+ξiξi≥0, ξi∗≥0, w⋅w≤cn
拉格朗日函数
L ( w , b , α , β , λ ) = ∑ i = 1 l ξ i + ∑ i = 1 l ξ i ∗ + ∑ i = 1 l β i ( − ξ i ) + ∑ i = 1 l β i ∗ ( − ξ i ∗ ) + λ 2 ( w ⋅ w − c n ) + ∑ i = 1 l α i ∗ ( y i − ( w ⋅ x i + b ) − ϵ − ξ i ∗ ) + ∑ i = 1 l α i ( ( w ⋅ x i + b ) − y i − ϵ − ξ i ) , α i ≥ 0 , α i ∗ ≥ 0 , β i ≥ 0 , β i ∗ ≥ 0 , λ ≥ 0 L(w, b, \alpha, \beta, \lambda) = \sum\limits_{i=1}^l \xi_i + \sum\limits_{i=1}^l \xi^*_i + \sum\limits_{i=1}^l \beta_i (-\xi_i) + \sum\limits_{i=1}^l \beta^*_i (-\xi^*_i) + \dfrac{\lambda}{2} (w \cdot w - c_n) \\+ \sum\limits_{i=1}^l \alpha^*_i (y_i - (w \cdot x_i +b) - \epsilon - \xi^*_i) + \sum\limits_{i=1}^l \alpha_i ((w \cdot x_i +b) - y_i - \epsilon - \xi_i) , \\ \alpha_i \ge 0,\ \ \alpha^*_i \ge 0,\ \ \beta_i \ge 0,\ \ \beta^*_i \ge 0,\ \ \lambda \ge 0 L(w,b,α,β,λ)=i=1∑lξi+i=1∑lξi∗+i=1∑lβi(−ξi)+i=1∑lβi∗(−ξi∗)+2λ(w⋅w−cn)+i=1∑lαi∗(yi−(w⋅xi+b)−ϵ−ξi∗)+i=1∑lαi((w⋅xi+b)−yi−ϵ−ξi),αi≥0, αi∗≥0, βi≥0, βi∗≥0, λ≥0
目标为 max α , β , λ min w , b , ξ L \max\limits_{\alpha, \beta, \lambda}\min\limits_{w, b, \xi}L α,β,λmaxw,b,ξminL,对拉格朗日函数求梯度得
∂ L ∂ w = 0 → w = ∑ i = 1 l α i ∗ − α i λ x i ∂ L ∂ b = 0 → ∑ i = 1 l ( α i ∗ − α i ) = 0 ∂ L ∂ ξ i = 0 → α i + β i = 1 ∂ L ∂ ξ i ∗ = 0 → α i ∗ + β i ∗ = 1 \dfrac{\partial L}{\partial w} = 0 \to w = \sum\limits_{i=1}^l \dfrac{\alpha^*_i - \alpha_i}{\lambda} x_i\\ \dfrac{\partial L}{\partial b} = 0 \to \sum\limits_{i=1}^l (\alpha^*_i - \alpha_i) = 0\\ \dfrac{\partial L}{\partial \xi_i} = 0 \to \alpha_i + \beta_i = 1\\ \dfrac{\partial L}{\partial \xi^*_i} = 0 \to \alpha^*_i + \beta^*_i = 1 ∂w∂L=0→w=i=1∑lλαi∗−αixi∂b∂L=0→i=1∑l(αi∗−αi)=0∂ξi∂L=0→αi+βi=1∂ξi∗∂L=0→αi∗+βi∗=1
代入拉格朗日函数可得到该问题的对偶问题,最大化泛函
W ( α , λ ) = − ϵ ∑ i = 1 l ( α i ∗ + α i ) + ∑ i = 1 l y i ( α i ∗ − α i ) − 1 2 λ ∑ i , j = 1 l ( α i ∗ − α i ) ( α j ∗ − α j ) ( x i ⋅ x j ) − λ 2 c n W(\alpha, \lambda) = -\epsilon \sum\limits_{i=1}^l (\alpha^*_i + \alpha_i) + \sum\limits_{i=1}^l y_i (\alpha^*_i - \alpha_i) - \dfrac{1}{2\lambda}\sum\limits_{i,j=1}^l (\alpha^*_i - \alpha_i) (\alpha^*_j - \alpha_j) (x_i \cdot x_j) - \dfrac{\lambda}{2} c_n W(α,λ)=−ϵi=1∑l(αi∗+αi)+i=1∑lyi(αi∗−αi)−2λ1i,j=1∑l(αi∗−αi)(αj∗−αj)(xi⋅xj)−2λcn
约束条件
∑ i = 1 l ( α i ∗ − α i ) = 0 , 0 ≤ α i ≤ 1 , 0 ≤ α i ∗ ≤ 1 , λ ≥ 0 , β = 1 − α \sum\limits_{i=1}^l (\alpha^*_i -\alpha_i) = 0,\ \ 0 \le \alpha_i \le 1,\ \ 0 \le \alpha^*_i \le 1,\ \ \lambda \ge 0,\ \ \ \beta = 1- \alpha i=1∑l(αi∗−αi)=0, 0≤αi≤1, 0≤αi∗≤1, λ≥0, β=1−α
还需满足 Kuhn-Tucker条件
β i ξ i = 0 , β i ∗ ξ i ∗ = 0 , λ ( w ⋅ w − c n ) = 0 α i ∗ ( y i − ( w ⋅ x i + b ) − ϵ − ξ i ∗ ) = 0 , α i ( ( w ⋅ x i + b ) − y i − ϵ − ξ i ) = 0 \beta_i \xi_i = 0,\ \ \ \beta^*_i \xi^*_i = 0,\ \ \ \lambda (w \cdot w - c_n) = 0\\ \alpha^*_i (y_i - (w \cdot x_i +b) - \epsilon - \xi^*_i) = 0,\ \ \ \alpha_i ((w \cdot x_i +b) - y_i - \epsilon - \xi_i) = 0 βiξi=0, βi∗ξi∗=0, λ(w⋅w−cn)=0αi∗(yi−(w⋅xi+b)−ϵ−ξi∗)=0, αi((w⋅xi+b)−yi−ϵ−ξi)=0
其中 α i ∗ − α i λ ≠ 0 \dfrac{\alpha^*_i - \alpha_i}{\lambda} \not = 0 λαi∗−αi=0 的项决定了支持向量( x i x_i xi 对 w w w 有贡献)。
采用软间隔的基本解
最小化泛函
Φ ( w , b ) = 1 2 w ⋅ w + C ( ∑ i = 1 l ξ i + ∑ i = 1 l ξ i ∗ ) \Phi(w, b) = \dfrac{1}{2} w \cdot w + C(\sum\limits_{i=1}^l \xi_i + \sum\limits_{i=1}^l \xi^*_i) Φ(w,b)=21w⋅w+C(i=1∑lξi+i=1∑lξi∗)
拉格朗日函数
L ( w , b , α , β ) = 1 2 w ⋅ w + C ∑ i = 1 l ξ i + C ∑ i = 1 l ξ i ∗ + ∑ i = 1 l β i ( − ξ i ) + ∑ i = 1 l β i ∗ ( − ξ i ∗ ) + ∑ i = 1 l α i ∗ ( y i − ( w ⋅ x i + b ) − ϵ − ξ i ∗ ) + ∑ i = 1 l α i ( ( w ⋅ x i + b ) − y i − ϵ − ξ i ) , α i ≥ 0 , α i ∗ ≥ 0 , β i ≥ 0 , β i ∗ ≥ 0 L(w, b, \alpha, \beta) = \dfrac{1}{2} w \cdot w + C\sum\limits_{i=1}^l \xi_i + C\sum\limits_{i=1}^l \xi^*_i + \sum\limits_{i=1}^l \beta_i (-\xi_i) + \sum\limits_{i=1}^l \beta^*_i (-\xi^*_i)\\ + \sum\limits_{i=1}^l \alpha^*_i (y_i - (w \cdot x_i +b) - \epsilon - \xi^*_i) + \sum\limits_{i=1}^l \alpha_i ((w \cdot x_i +b) - y_i - \epsilon - \xi_i) ,\\ \alpha_i \ge 0,\ \ \alpha^*_i \ge 0,\ \ \beta_i \ge 0,\ \ \beta^*_i \ge 0 L(w,b,α,β)=21w⋅w+Ci=1∑lξi+Ci=1∑lξi∗+i=1∑lβi(−ξi)