ML step by step|20200103

Machine Learning problem discussion

Proof that min of the regularized error function is equivalent to minimizing the unregularized sum-of-squares error

(2 points) Using the technique of Lagrange multipliers, show that minimization of the regularized error function
$$\frac{1}{2}\sum _{i=1}^n{\left(y_i-{\omega }^T{\mathbf{x}}_i\right)}^{{}^2}+\frac{\lambda }{2}\sum _{i=1}^n{\left|{\omega }_i\right|}^{{}^q}$$
is equivalent to minimizing the unregularized sum-of-squares error
$$\frac{1}{2}\sum _{i=1}^n{\left(y_i-{\omega }^T{\mathbf{x}}_i\right)}^{{}^2}$$
subject to the constraint
$$\sum _{i=1}^n{\left|{\omega }_j\right|}^{{}^q}⩽\eta$$

Proof :
$$\begin{gathered} min\frac{1}{2}\sum _{i=1}^n{\left(y_i-{\omega }^T{\mathbf{x}}_i\right)}^{{}^2} \\ s.t\sum _{i=1}^n{\left|{\omega }_j\right|}^{{}^q}⩽\eta \end{gathered}$$
It is a convex optimization problem. Let’s write its Lagrange function.
$$L=\sum _{i=1}^n{\left(y_i-{\omega }^T{\mathbf{x}}_i\right)}^{{}^2}+\lambda ({\left|{\omega }_j\right|}^{{}^q}-\eta )$$
According to KKT conditions, and let w* , lambda* be the optimal solution for original and dual problem.
$$0⩽{\lambda }^{\ast }$$

$$\begin{gathered} 0={▽}_{\omega }(\sum _{i=1}^n{\left(y_i-{\omega }^T{\mathbf{x}}_i\right)}^{{}^2}+\lambda ({\left|{\omega }_j\right|}^{{}^q}-\eta )) \\ =\sum _{i=1}^n(-x_i)(y_i-\sum _{i=1}^n{\omega }_j{x}_{ij})+\lambda q{\left|{\omega }_j\right|}^{{}^{q-1}}{k}_j \\ {k}_j=\left\{\begin{array}{c}\subseteq [-1,1],if{\omega }_j=0\\ 1,if{\omega }_j>0\\ -1,if{\omega }_j<0\end{array}\right\} \end{gathered}$$

$$\therefore \lambda q{\left|{\omega }_j\right|}^{{}^{q-1}}{k}_j=\sum _{i=1}^n\left((-{\lambda }_i)(y_i-\sum _{j=1}^n{\omega }_j{x}_{ij})\right)$$

For the unconstrained regularized optimization function,

$$\begin{gathered} F.O.C\frac{\partial }{\partial x}\left[\frac{1}{2}\sum _{i=1}^n{\left(y_i-{\omega }^T{\mathbf{x}}_i\right)}^{{}^2}+\frac{\lambda }{2}\sum _{j=1}^n\left|{\omega }^q\right|\right]=-\sum _{i=1}^n{x}_{ij}(y_i-\sum _{j=1}^n{x}_{ij}{\omega }_j)+\frac{\lambda }{2}\sum _{j=1}^n{\left|\omega \right|}^{q-1}{k}_j=0 \\ \therefore \frac{\lambda }{2}q{\left|\omega \right|}^{q-1}{k}_j=\sum _{i=1}^n\left((-{\lambda }_i)(y_i-\sum _{j=1}^n{\omega }_j{x}_{ij})\right) \end{gathered}$$

MAP of LASSO

(2 points) (MAP Estimation) We mentioned that when θ is an isotropic Laplace distribution, MAP corresponds to LASSO (L1-regularization). Now you are maximizing the likelihood function ${\prod }_{i=1}^{n}p(x_i|\theta )$ with prior distribution p(θ).
$$p(\theta )=\frac{\lambda }{2}exp(-\lambda \left|\theta \right|)\lambda >0$$
Please prove that it is equivalent to maximizing
$$log\prod _{i=1}^{n}p(x_i|\theta )-\lambda \left|\theta \right|$$
**Proof : **
$$\begin{gathered} argmax\theta (\prod _{i=1}^{n}p(x_i|\theta )p(\theta )) \\ =argmax\theta log(\prod _{i=1}^{n}p(x_i|\theta )p(\theta ))) \\ =argmax\theta log(\prod _{i=1}^{n}p(x_i|\theta ))+logp(\theta ) \\ =argmax\theta log(\prod _{i=1}^{n}p(x_i|\theta ))+log\frac{1}{2}\lambda -\lambda \left|\theta \right| \\ =argmax\theta log(\prod _{i=1}^{n}p(x_i|\theta ))-\lambda \left|\theta \right| \end{gathered}$$

Bias-Variance Tradeoff and its applications

(2 points) (Mean Square Error) We mentioned Bias-Variance Tradeoff in class. We define the MSE of $\hat{X}$ an estimator of X as
$$MSE(X^{)}\triangleq E[(X^{-}X{)}^{{}^2}]$$
The variance of X^ is defined as
$$Var(\hat{X})\triangleq E[(\hat{X}-E[\hat{X}]{)}^{{}^2}]$$
and the bias is defined as
$$Bias(\hat{X})\triangleq E[\hat{X}]-X$$.

(a) Please prove that
$$MSE[\hat{X}]=Var[\hat{X}]+(Bais[\hat{X}]{)}^{{}^2}$$
(b) Our data are added with an independent Gaussian noise, say, X + N, where E[N] = 0 and E[N2] = σ2 and the estimator is X^. We define the empirical MSE as $E[(\hat{X}-X-N{)}^{{}^2}]$.

Please prove that
$$E[(\hat{X}-X-N{)}^{{}^2}]=MSE[\hat{X}]+{\sigma }^2$$
The equation tells us that the empirical error is a good estimation of the true error. Thus, we can minimize the empirical error in order to properly minimize the true error.

Proof :

(a)
$$\begin{gathered} MSE[\hat{X}]=E[[\hat{X}-X{]}^{{}^2}] \\ =E[[(\hat{X}-E(\hat{X})+(X+E(\hat{X}){]}^{{}^2}] \\ =E[(\hat{X}-E(\hat{X}{)}^{{}^2}]+E[(X-E(\hat{X}{)}^{{}^2}]-2E[(\hat{X}-E(\hat{X})(X-E(\hat{X})] \\ =Var[\hat{X}]+(Bais[\hat{X}]{)}^{{}^2}-2E[\hat{X}X+E(\hat{X})E(\hat{X})-XE(\hat{X})-\hat{X}E(X))]= \\ =Var[\hat{X}]+(Bais[\hat{X}]{)}^{{}^2} \end{gathered}$$
(b)
$$\begin{gathered} E[(\hat{X}-X-N{)}^{{}^2}]=E[{\hat{X}}^{{}^2}]+E[(\hat{X}+N{)}^{{}^2}]-2E[\hat{X}(X+N)] \\ =Var[\hat{X}]+E[\hat{X}{]}^{{}^2}-2E[\hat{X}X]-2E[\hat{X}N]+E[(X+N{)}^{{}^2}] \\ =Var[\hat{X}]+E[\hat{X}{]}^{{}^2}+E[X^{{}^2}]+E[N^{{}^2}]+2E[XN]-2E[\hat{X}X]-2E[\hat{X}N] \\ =Var[\hat{X}]+E[(\hat{X}-X{)}^{{}^2}]+{\sigma }^2 \\ =MSE[\hat{X}]+{\sigma }^2 \end{gathered}$$

VC Dimension’s application

(4 points) (VC Dimension) Given some finite domain set, $\chi$, and a number $k\le \chi$ please figure out the VC-dimension of each of the following classes:
(a) (2 points)
H κ χ = { h ∈ { 0 , 1 } κ : ∣ { x : h ( x ) = 1 } ∣ = k } \Eta _{\kappa }^{\chi }=\left \{ h\in {\left\{0,1\right\}^{\kappa}:| \left \{ x:h(x)=1 \right \}|=k } \right \} Hκχ​={h∈{0,1}κ:∣{x:h(x)=1}∣=k}
That is, the set of all functions that assign the value 1 to exactly k elements of $\kappa$.
(b) (2 points)
H κ χ = { h ∈ { 0 , 1 } κ : ∣ { x : h ( x ) = 0 } ∣ ≤ k    o r    { x : h ( x ) = 1 } ∣ ≤ k } \Eta _{\kappa }^{\chi }=\left \{ h\in {\left\{0,1\right\}^{\kappa}:| \left \{ x:h(x)=0 \right \}|\leq k ~~or~~\left \{ x:h(x)=1 \right \}|\leq k } \right \} Hκχ​={h∈{0,1}κ:∣{x:h(x)=0}∣≤k  or  {x:h(x)=1}∣≤k}
Solution:

(a)

Since for every hypothesis class, there are exactly k elements of $\chi$ to make $h(x)=1$. It means when we give k+1 points as a subset to shatter, and labeled all of them with “1”, then we can’t find a hypothesis from the above classes to fit them. Then for any k points or any less than k points set, for any labeling, we always can find a hypothesis to shatter them. Hence, the VC dimension of the ${\mathrm{H}}_k^{\chi }$ is k. However, another case, if you think $\left|\chi \right|$ is not large enough, when k points in the whole set is marked as 1, then $\left|\chi \right|-k$ is marked as 0. Hence, there is a labeling(mark all negative) which is not able to shatter. Then, the VC-dimensions will be $\left|\chi \right|-k$. Hence, according to the above analysis, the VC-dimension will be $min(k,\left|\chi \right|-k)$.

(b)

Since for every hypothesis class, there are at most k elements of $\chi$ to make $h(x)=1orh(x)=0$. It means when we give a set of points in $\chi$ , and if there always exists $m$ points labeled 1 or 0 when $m<k$, then we always can find a hypothesis from the above classes to fit them. Hence, for any points with size less than $2\chi +1$ , we can shatter them with the hypothesis class. However, for $2\chi +2$ points, for example, mark $k+1$ points as 1 and another $k+1$ points 0, then no valid hypothesis in this $\mathrm{H}$ . Hence, the VC-dimension of the ${\mathrm{H}}_{\kappa }^{\chi }$ is $2k+1$. If considering the $\left|\chi \right|$ , it should be $min(2k+1,\left|\chi \right|)$.

reference

欢迎关注二幺子的知识输出通道：