The solution on the Elements of Statistical Learning

Preface

If you find any errata or have a good idea, please contact me via tongust@163.com.

Ex. 8

Ex 8.1

First of all, we should refer to the theory Kullback-Leibler Divergence. Here, I just give a brief derivation.
To proof KL divergence, we use the Jensen Inequality:

$$\int p(x)f(x)dx \ge f\{\int p(x)xdx\}......(1)$$
The constrains which the formula must satisfy are: the function $f(x)$ must be a convex function at convex set.
In this case, we construct a simple convex function $f(x)=-ln(x)$, and it has the following property:
$$-\int p(x)ln(x)dx \ge -ln\{\int p(x)xdx\}......(2)$$
Substitute $x$ = $\frac{q(x)}{p(x)}$ into (2):
$$-\int p(x)ln(\frac{q(x)}{p(x)})dx \ge -ln\{\int p(x)\frac{q(x)}{p(x)}dx\}......(3)$$
$$\int p(x)ln(\frac{q(x)}{p(x)})dx \le ln\{\int q(x)dx\}......(4)$$
As we know $q(x)$ is a distribution function, it is obvious that $\int q(x)dx = 1$.
Therefore, we get the KL divergence:
$$D_{KL}(p||q) = \int p(x)ln(\frac{q(x)}{p(x)})dx \le 0......(5)$$
Since we has shown that (8.61) is maximized as a function of $r(y)$ when $r(y) = q(y)$, $-R(\theta',\theta)=-\int p(Z|\theta)ln(Z^m|Z, \theta',\theta)$ is a convex function and satisfies KL divergence. Hence $R(\theta',\theta) - R(\theta,\theta) \le 0$
$\square$

Ex. 8.2

Off the topics

Since this exercise is based on one paper$^{[1]}$, I would say that our lovely authors of ESL just overestimated those poor readers like me to be excellent mathematicians.;)

Proof

I will use the notations in [1] instead of those in ESL which is a bit confusing to me. (Denote $Z^m$ as $y$)
We want to proof : For a fixed value $\theta$, there is a unique distribution, $P_{\theta}$, given by $P_{\theta}(y)=P(y|z,\theta)$, which maximizes the log-likelihood (8.48).
From the hint, we use Lagrange Multiplier to rewrite our formula.

$$L({P}(y), \lambda) =$$ $$-\sum_{y=1}^{n}P(y_i)ln(P_{\theta}(z, y))+\sum_{y_i}^{n}P(y_i)ln(P(y_i))+\lambda(1-\sum_{y=1}^{n}P(y_i))......(1)$$
To get the stationary points, we set the gradient of $L(P(y), \lambda)$ W.R.T (with respect to) $P(y_i) (i = 1,2,...,n)$ with zero :
$$\frac{\mathrm{d}L}{\mathrm{d}P(y_i)} = -lnP_{\theta}(z,y) +1+lnP(y_i)-\lambda =0......(2)$$
To simplify:
$$\lambda=1-ln(\frac{P(y_i))}{P(z,y|\theta)})......(3)$$
$$P(y_i)=exp(1-\lambda)P(z,y|\theta)......(4)$$
$$i = 1,2,3,...,n$$
From (4), it follows that $P(y)$ must be proportional to $P_{\theta}(z,y)=P(y,z|\theta)$. Also we notice that $\sum_y P(y)= 1$
Summing (4) W.R.T $y_i$, we can see:
$$1=\sum_yP(y)=exp(1-\lambda)\sum_yP(y,z|\theta)......(5)$$
$$\sum_yP(y,z|\theta)=P(z|\theta)=\frac{1}{exp(1-\lambda)}......(6)$$
$$exp(1-\lambda)=\frac{1}{P(z|\theta)}......(7)$$
Substitute (7) into (4):
$$P(y_i)=\frac{P(z,y|\theta)}{P(z|\theta)}=P(y|z,\theta)......(8)$$
$\square$

Ex. 8.3

Ex. 8.4

Ex. 8.5

Ex. 8.6

Ex. 8.7

Proof f(x) is non-decreasing under update (8.63)

From (8.62), we have

$$f(x^{s+1}) \ge g(x^{s+1}, x^s)\ge g(x^s, x^s) = f(x^s) ......(1)$$
$\square$

Proof EM algorithm (Sec. 8.5.2) is an example of an EM algorithms

This exercise need us to show following:

$$Q(\theta', \theta) + log(Pr(\bf{Z}|\theta))-Q(\theta, \theta) \le log(\theta', \bf{Z} )......(2)$$
On one hand, from (8.46), we can denote that:
$$log(Pr(\bf{Z}|\theta)) = Q(\theta, \theta)-R(\theta, \theta)......(3)$$
Hence, the left hand side (l.h.s) of equation (2) can be simplified as:
$$Q(\theta', \theta) +Q(\theta, \theta)-R(\theta, \theta)-Q(\theta, \theta) = (\theta', \theta) - R(\theta', \theta)......(4)$$
On the other hand, also from (8.46), the r.h.s of (2) can be written as:
$$log(\theta', \bf{Z}) = Q(\theta', \theta) -R(\theta', \theta)......(5)$$
From Ex. 8.1, we see:
$$R(\theta, \theta) \ge R(\theta', \theta).......(6)$$
$$-R(\theta, \theta) \le -R(\theta', \theta).......(7)$$
$$Q(\theta', \theta)-R(\theta, \theta) \le Q(\theta', \theta) -R(\theta', \theta).......(8)$$
Finally, we get (4) $\le$ (5) to finish our demonstration.
$\square$

Reference

[1] Neal, Radford M., and G. E. Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants. Learning in Graphical Models. Springer Netherlands, 2000:355-368.

Ex 15 Random Forest

15.1

对于bagging我们有B个trees，这些trees是identical distributed 而不是，i.i.d (independent identical distributed data). 这里的trees是correlated, $\rho = \frac{Var(x_ix_j)}{\sigma^2}$，因为$x_i$是同分布的，所以其$\sigma$是identical.

$$y = \frac{1}{B}\sum_{i=1}^{B}x_i\\ E[y] = Ex\\ Var[y] = E[y^2]-E^2[x] = \frac{1}{B^2}E[\sum_{i=1}^{B}x_i^2+\sum_{i\ne j}x_ix_j] - E^2[x]\\ = \frac{B}{B^2}E[x^2]+\frac{B^2-B}{B^2}E_{i\ne j}[x_ix_j] - E^2[x]............(1)$$

1. $\rho=0$

$$\frac{Var[(x_i-E[x])(x_j-E[x])]}{\sigma^2} = \frac{E[x_ix_j]-E^2[x]}{\sigma^2} = 0\\ ⇒ E_{i\ne j}[x_ix_j] = E^2[x]..................(2)$$
(2) $\to$(1):
$$Var[y]=\frac{1}{B}(\frac{BE[x^2]+(B^2-B)E^2[x]-B^2E^2[x]}{B^2})=\frac{1}{B}\sigma^2$$

2. $\rho > 0$

$$E[x_ix_j]=\sigma^2\rho + E^2[x]\\ Var[y] = \frac{B}{B^2}E^2[x]+\frac{B^2-B}{B^2}(\sigma^2\rho+E^2[x])-E^2[x]\\ Var[y]=\frac{1}{B}\sigma^2+\rho\sigma^2-\frac{1}{B}\rho\sigma^2\\ $$
$\square$