# ML step by step|20200103

## Machine Learning problem discussion

### Proof that min of the regularized error function is equivalent to minimizing the unregularized sum-of-squares error

(2 points) Using the technique of Lagrange multipliers, show that minimization of the regularized error function
1 2 ∑ i = 1 n ( y i − ω T x i ) 2 + λ 2 ∑ i = 1 n ∣ ω i ∣ q \frac{1}{2}\sum_{i=1}^{n}\left ( y_{i}-\mathbf{\omega}^{T}\mathbf{x}_{i}\right )^{^{2}}+\frac{\lambda }{2}\sum_{i=1}^{n}\left | \omega_i \right |^{^{q}}
is equivalent to minimizing the unregularized sum-of-squares error
1 2 ∑ i = 1 n ( y i − ω T x i ) 2 \frac{1}{2}\sum_{i=1}^{n}\left ( y_{i}-\mathbf{\omega}^{T}\mathbf{x}_{i}\right )^{^{2}}
subject to the constraint
∑ i = 1 n ∣ ω j ∣ q ⩽ η \sum_{i=1}^{n}\left | \omega_j \right |^{^{q}}\leqslant \eta

Proof :
m i n 1 2 ∑ i = 1 n ( y i − ω T x i ) 2 s . t ∑ i = 1 n ∣ ω j ∣ q ⩽ η min \frac{1}{2}\sum_{i=1}^{n}\left ( y_{i}-\mathbf{\omega}^{T}\mathbf{x}_{i}\right )^{^{2}}\\s.t\sum_{i=1}^{n}\left | \omega_j \right |^{^{q}}\leqslant \eta
It is a convex optimization problem. Let’s write its Lagrange function.
L = ∑ i = 1 n ( y i − ω T x i ) 2 + λ ( ∣ ω j ∣ q − η ) L =\sum_{i=1}^{n}\left ( y_{i}-\mathbf{\omega}^{T}\mathbf{x}_{i}\right )^{^{2}}+\lambda (\left | \omega_j \right |^{^{q}}-\eta)\\
According to KKT conditions, and let w* , lambda* be the optimal solution for original and dual problem.
0 ⩽ λ ∗ 0\leqslant \lambda^*

0 = ▽ ω ( ∑ i = 1 n ( y i − ω T x i ) 2 + λ ( ∣ ω j ∣ q − η ) ) = ∑ i = 1 n ( − x i ) ( y i − ∑ i = 1 n ω j x i j ) + λ q ∣ ω j ∣ q − 1 k j k j = { ⊆ [ − 1 , 1 ] , i f    ω j = 0 1 , i f    ω j > 0 − 1 , i f    ω j < 0 } 0=\bigtriangledown _{\omega}(\sum_{i=1}^{n}\left ( y_{i}-\mathbf{\omega}^{T}\mathbf{x}_{i}\right )^{^{2}}+\lambda (\left | \omega_j \right |^{^{q}}-\eta))\\=\sum_{i=1}^{n}(-x_i)(y_i-\sum_{i=1}^{n}\omega_jx_{ij})+\lambda q\left | \omega_j \right |^{^{q-1}}k_j\\ k_j=\begin{Bmatrix} \subseteq [-1,1], if ~~\omega_j=0 \\ 1, if ~~\omega_j>0 \\ -1, if ~~\omega_j<0 \end{Bmatrix}

∴ λ q ∣ ω j ∣ q − 1 k j = ∑ i = 1 n ( ( − λ i ) ( y i − ∑ j = 1 n ω j x i j ) ) \therefore \lambda q\left | \omega_j \right |^{^{q-1}}k_j= \sum_{i=1}^{n}\left ((-\lambda_i)(y_{i}-\sum_{j=1}^{n}\omega_jx_{ij})\right)

For the unconstrained regularized optimization function,

F . O . C     ∂ ∂ x [ 1 2 ∑ i = 1 n ( y i − ω T x i ) 2 + λ 2 ∑ j = 1 n ∣ ω q ∣ ] = − ∑ i = 1 n x i j ( y i − ∑ j = 1 n x i j ω j ) + λ 2 ∑ j = 1 n ∣ ω ∣ q − 1 k j = 0 ∴ λ 2 q ∣ ω ∣ q − 1 k j = ∑ i = 1 n ( ( − λ i ) ( y i − ∑ j = 1 n ω j x i j ) ) F.O.C ~~~\frac{\partial }{\partial x} \left [ \frac{1}{2}\sum_{i=1}^{n}\left ( y_{i}-\mathbf{\omega}^{T}\mathbf{x}_{i}\right )^{^{2}} + \frac{\lambda }{2}\sum_{j=1}^{n}\left | \mathbf{\omega }^{q} \right | \right ] =-\sum_{i=1}^{n}x_{ij}(y_i-\sum_{j=1}^{n}x_{ij}\omega_j)+\frac{\lambda }{2}\sum_{j=1}^{n}\left | \mathbf{\omega } \right |^{q-1}k_j=0\\ \therefore \frac{\lambda }{2}q\left | \mathbf{\omega } \right |^{q-1}k_j =\sum_{i=1}^{n}\left ((-\lambda_i)(y_{i}-\sum_{j=1}^{n}\omega_jx_{ij})\right)

### MAP of LASSO

(2 points) (MAP Estimation) We mentioned that when θ is an isotropic Laplace distribution, MAP corresponds to LASSO (L1-regularization). Now you are maximizing the likelihood function ∏ i = 1 n p ( x i ∣ θ ) \prod_{i=1}^{n}p(x_i|\theta) with prior distribution p(θ).
p ( θ ) = λ 2 e x p ( − λ ∣ θ ∣ )        λ > 0 p(\theta )= \frac{\lambda }{2}exp(-\lambda\left |\theta \right |)~~~~~~\lambda>0\\
Please prove that it is equivalent to maximizing
l o g ∏ i = 1 n p ( x i ∣ θ ) − λ ∣ θ ∣ log\prod_{i=1}^{n}p(x_i|\theta)-\lambda\left |\theta \right |
**Proof : **
a r g m a x θ ( ∏ i = 1 n p ( x i ∣ θ ) p ( θ ) ) = a r g m a x θ l o g ( ∏ i = 1 n p ( x i ∣ θ ) p ( θ ) ) ) = a r g m a x θ l o g ( ∏ i = 1 n p ( x i ∣ θ ) ) + l o g p ( θ ) = a r g m a x θ l o g ( ∏ i = 1 n p ( x i ∣ θ ) ) + l o g 1 2 λ − λ ∣ θ ∣ = a r g m a x θ l o g ( ∏ i = 1 n p ( x i ∣ θ ) ) − λ ∣ θ ∣ argmax \theta (\prod_{i=1}^{n}p(x_i|\theta)p(\theta)) \\=argmax\theta log(\prod_{i=1}^{n}p(x_i|\theta)p(\theta))) \\=argmax\theta log(\prod_{i=1}^{n}p(x_i|\theta))+logp(\theta) \\=argmax\theta log(\prod_{i=1}^{n}p(x_i|\theta))+log\frac{1}{2}\lambda-\lambda\left | \theta \right | \\=argmax\theta log(\prod_{i=1}^{n}p(x_i|\theta))-\lambda\left | \theta \right |

### Bias-Variance Tradeoff and its applications

(2 points) (Mean Square Error) We mentioned Bias-Variance Tradeoff in class. We define the MSE of X ^ \hat{X} an estimator of X as
M S E ( X ) ≜ E [ ( X − X ) 2 ] MSE(X^) ≜ E[(X^-X)^{^{2}}]
The variance of X^ is defined as
V a r ( X ^ ) ≜ E [ ( X ^ − E [ X ^ ] ) 2 ] Var(\hat{X}) ≜ E[(\hat{X} -E[\hat{X}])^{^{2}}]
and the bias is defined as
B i a s ( X ^ ) ≜ E [ X ^ ] − X Bias(\hat{X}) ≜ E[\hat{X}]-X .

M S E [ X ^ ] = V a r [ X ^ ] + ( B a i s [ X ^ ] ) 2 MSE[\hat{X}]=Var[\hat{X}]+(Bais[\hat{X}])^{^{2}}
(b) Our data are added with an independent Gaussian noise, say, X + N, where E[N] = 0 and E[N2] = σ2 and the estimator is X^. We define the empirical MSE as E [ ( X ^ − X − N ) 2 ] E[(\hat{X} - X - N)^{^2}] .

E [ ( X ^ − X − N ) 2 ] = M S E [ X ^ ] + σ 2 E[(\hat{X} - X - N)^{^2}]=MSE[\hat{X}]+\sigma ^{2}
The equation tells us that the empirical error is a good estimation of the true error. Thus, we can minimize the empirical error in order to properly minimize the true error.

Proof :

(a)
M S E [ X ^ ] = E [ [ X ^ − X ] 2 ] = E [ [ ( X ^ − E ( X ^ ) + ( X + E ( X ^ ) ] 2 ] = E [ ( X ^ − E ( X ^ ) 2 ] + E [ ( X − E ( X ^ ) 2 ] − 2 E [ ( X ^ − E ( X ^ ) ( X − E ( X ^ ) ] = V a r [ X ^ ] + ( B a i s [ X ^ ] ) 2 − 2 E [ X ^ X + E ( X ^ ) E ( X ^ ) − X E ( X ^ ) − X ^ E ( X ) ) ] = = V a r [ X ^ ] + ( B a i s [ X ^ ] ) 2 MSE[\hat{X}]=E[[\hat{X}-X]^{^{2}}] \\=E[[(\hat{X}-E(\hat{X})+(X+E(\hat{X})]^{^{2}}] \\=E[(\hat{X}-E(\hat{X})^{^{2}}]+E[(X-E(\hat{X})^{^{2}}]-2E[(\hat{X}-E(\hat{X})(X-E(\hat{X})] \\=Var[\hat{X}]+(Bais[\hat{X}])^{^{2}}-2E[\hat{X}X+E(\hat{X})E(\hat{X})-XE(\hat{X})-\hat{X}E(X))] =\\=Var[\hat{X}]+(Bais[\hat{X}])^{^{2}}
(b)
E [ ( X ^ − X − N ) 2 ] = E [ X ^ 2 ] + E [ ( X ^ + N ) 2 ] − 2 E [ X ^ ( X + N ) ] = V a r [ X ^ ] + E [ X ^ ] 2 − 2 E [ X ^ X ] − 2 E [ X ^ N ] + E [ ( X + N ) 2 ] = V a r [ X ^ ] + E [ X ^ ] 2 + E [ X 2 ] + E [ N 2 ] + 2 E [ X N ] − 2 E [ X ^ X ] − 2 E [ X ^ N ] = V a r [ X ^ ] + E [ ( X ^ − X ) 2 ] + σ 2 = M S E [ X ^ ] + σ 2 E[(\hat{X} - X - N)^{^2}]=E[\hat{X}^{^{2}}]+E[(\hat{X}+N)^{^{2}}]-2E[\hat{X}(X+N)] \\=Var[\hat{X}]+E[\hat{X}]^{^{2}}-2E[\hat{X}X]-2E[\hat{X}N]+E[(X+N)^{^2}] \\=Var[\hat{X}]+E[\hat{X}]^{^{2}}+E[X^{^{2}}]+E[N^{^{2}}]+2E[XN]-2E[\hat{X}X]-2E[\hat{X}N] \\=Var[\hat{X}]+E[(\hat{X}-X)^{^2}]+\sigma ^{2} \\=MSE[\hat{X}]+\sigma ^{2}

### VC Dimension’s application

(4 points) (VC Dimension) Given some finite domain set, χ \chi , and a number k ≤ χ k ≤ \chi please figure out the VC-dimension of each of the following classes:
(a) (2 points)
H κ χ = { h ∈ { 0 , 1 } κ : ∣ { x : h ( x ) = 1 } ∣ = k } \Eta _{\kappa }^{\chi }=\left \{ h\in {\left\{0,1\right\}^{\kappa}:| \left \{ x:h(x)=1 \right \}|=k } \right \}
That is, the set of all functions that assign the value 1 to exactly k elements of κ \kappa .
(b) (2 points)
H κ χ = { h ∈ { 0 , 1 } κ : ∣ { x : h ( x ) = 0 } ∣ ≤ k    o r    { x : h ( x ) = 1 } ∣ ≤ k } \Eta _{\kappa }^{\chi }=\left \{ h\in {\left\{0,1\right\}^{\kappa}:| \left \{ x:h(x)=0 \right \}|\leq k ~~or~~\left \{ x:h(x)=1 \right \}|\leq k } \right \}
Solution:

(a)

Since for every hypothesis class, there are exactly k elements of χ \chi to make h ( x ) = 1 h(x)=1 . It means when we give k+1 points as a subset to shatter, and labeled all of them with “1”, then we can’t find a hypothesis from the above classes to fit them. Then for any k points or any less than k points set, for any labeling, we always can find a hypothesis to shatter them. Hence, the VC dimension of the H k χ \Eta_{k}^{\chi} is k. However, another case, if you think ∣ χ ∣ \left |\chi \right | is not large enough, when k points in the whole set is marked as 1, then ∣ χ ∣ − k \left |\chi \right |-k is marked as 0. Hence, there is a labeling(mark all negative) which is not able to shatter. Then, the VC-dimensions will be ∣ χ ∣ − k \left |\chi \right |-k . Hence, according to the above analysis, the VC-dimension will be m i n ( k , ∣ χ ∣ − k ) min(k,\left |\chi \right |-k) .

(b)

Since for every hypothesis class, there are at most k elements of χ \chi to make h ( x ) = 1   o r   h ( x ) = 0 h(x)=1~or~h(x)=0 . It means when we give a set of points in χ \chi , and if there always exists m m points labeled 1 or 0 when m < k m<k , then we always can find a hypothesis from the above classes to fit them. Hence, for any points with size less than 2 χ + 1 2\chi+1 , we can shatter them with the hypothesis class. However, for 2 χ + 2 2\chi+2 points, for example, mark k + 1 k+1 points as 1 and another k + 1 k+1 points 0, then no valid hypothesis in this H \Eta . Hence, the VC-dimension of the H κ χ \Eta _{\kappa }^{\chi } is 2 k + 1 2k+1 . If considering the ∣ χ ∣ \left |\chi \right | , it should be m i n ( 2 k + 1 , ∣ χ ∣ ) min(2k+1,\left |\chi \right |) .

reference

• 点赞
• 评论
• 分享
x

海报分享

扫一扫，分享海报

• 收藏
• 打赏

打赏

天天学习的零柒贰幺

你的鼓励将是我创作的最大动力

C币 余额
2C币 4C币 6C币 10C币 20C币 50C币
• 举报
• 一键三连

点赞Mark关注该博主, 随时了解TA的最新博文