Stanford 机器学习课程cs229 数学推导知识

if $x$ is a row vector,then

$$x^Tx=(x \cdot x)=\lVert x \rVert^2_2=tr(x^Tx)$$

Linear regression

if $A$ and $B$ are square matrices, and $a$ is a real number:

$$\begin{align} trABC &= trCAB=trBCA, \\ trABCD &= trDABC=trCDAB=trBCDA. \\ trA &=trA^T \\ tr(A+B) &= trA+trB \\ tr \, aA &=atrA \\ \nabla_A trAB &= B^T \\ \nabla_{A^T}f(A) &= (\nabla_Af(A))^T \\ \nabla_{A^T} trAB &= B \\ \nabla_A trABA^TC &= CAB+C^TAB^T \\ \nabla_{A^T} trABA^TC &= B^TA^TC^T+BA^TC \\ if \; C=I \; then \quad \nabla_A trABA^T &=B^TA^T+BA^T \\ \nabla_A |A| &= |A|(A^{-1})^T \\ \nabla_\theta J(\theta) &= \nabla_\theta \frac 12 (X \theta-\vec y)^T(X \theta-\vec y) \\ &= \frac 12 \nabla_\theta \;(\theta^T X^T X \theta-\theta^T X^T \vec y-\vec y^T X \theta +\vec y^T \vec y)\\ &= \frac 12 \nabla_\theta \, tr \;(\theta^T X^T X \theta-\theta^T X^T \vec y-\vec y^T X \theta +\vec y^T \vec y) \\ &= \frac 12 \nabla_\theta \; (tr \, \theta^T X^T X \theta - 2 tr \, \vec y^T X \theta)\\ &= \frac 12 (X^T X \theta + X^T X \theta - 2X^T \vec y)\\ &= X^T X \theta-X^T \vec y \\ X^T X\theta &=X^T \vec y& \\ \theta &=(X^T X)^{-1}X^T \vec y \end{align}$$

Locally weighted linear regression

$$\begin{align} w^{(i)} = exp \left( - { (x^{(i)}-x)^2 \over 2\tau^2 } \right)\\ X^T W X \theta=X^T W \vec y \\ \theta=(X^T W X)^{-1}X^T W \vec y \\ \end{align}$$

Newton’s method:

$$\begin{align} \theta := \theta - H^{-1} \nabla_{\theta} \ell (\theta). \\ H_{ij}={ \partial^2 \ell(\theta) \over \partial \theta_i \partial \theta_j }. \end{align}$$
fit logistic regression using locally weighted lr:
the log-likelihood function for logistic regression:
$$\ell(\theta)= \sum_{i=1}^m y^{(i)} \log h(x^{(i)})+(1 - y^{(i)}) \log (1 - h(x^{(i)}))$$
for any vector $z$, it holds true that
$$\begin{align} z^T H z & \le 0. \\ { \partial \ell (\theta) \over \partial \theta_k } & = \sum_{i=1}^m(y^{(i)} - h(x^{(i)})) x_k^{(i)}\\ H_{kl} & = {\partial^2 \ell (\theta)} \\ & = \sum_{i=1}^m -{ \partial h(x^{(i)}) \over \partial \theta_l } x_k^{(i)} \\ & = \sum_{i=1}^m -h(x^{(i)})(1-h(x^{(i)})) x_l^{(i)} x_k^{(i)} \end{align}$$

The Exponential family

we say that a class of distribution is in the exponential family if it can be written in the form

$$\begin{align} p(y;\eta)=b(y)exp(\eta^T T(y)-a(\eta))). \end{align}$$
$Jensen's inequality$
Suppose we start with the inequality in the basic definition of a convex function
$$f(\theta x+(1-\theta)y) \le \theta f(x)+(1-\theta)f(y)\quad for \quad 0 \le \theta \le 1.$$
Using induction, this can be fairly easily extended to convex combinations of more than one point,
$$f \left( \sum_{i=1}^k \theta_i x_i \right) \le \sum_{i=1}^k \theta_i f(x_i) \quad for \quad \sum_{i=1}^k \theta_i =1,\; \theta_i \ge 0 \;\; \forall i.$$
In fact, this can also be extended to infinite sums or integrals. In the latter case, the inequality can be written as
$$f\left( \int p(x)xdx \right)\le \int p(x)f(x)dx \quad for \quad \int p(x)dx=1, \; p(x) \ge0 \;\; \forall x.$$
Because p(x) integrates to 1, it is common to consider it as a probability density, in which case the previous equation can be written in terms of expectations,
$$f(E[x]) \le E[f(x)]$$

Learning theory
Lemma. (The union bound). Let $A_1,A_2,\dots,A_k$ be $k$ different events (that may not be independent). Then

$$P( A_1 \cup \dots \cup A_k ) \le P(A_1) + \dots + P(A_k)$$
Lemma. (Hoeffding inequality) Let $Z_1,\dots,Z_m$ be $m$ independent and identically distributed (iid) random variables drawn from a $Bernoulli(\phi)$ distribution. Let $\hat{\phi}=(1/m)\sum_{i=1}^m Z_i$ be the mean of these random variables, and let any $\gamma \gt 0$ be fixed. Then

$$P(|\phi-\hat{\phi} \gt \gamma|) \; \le \; 2exp(-2 \gamma^2 m)$$
This lemma (which in learning theory is also called the Chernoff bound) says that if we take $\hat{\phi}-$ the average of $m \; \; Bernoulli(\phi)$ random variables $-$to be our estimate of $\phi$, then the probability of our being far from the true value is small, so long as $m$ is large.

$$\hat{h}= \arg \min_{h \in \mathcal{H}} \hat{\epsilon} (h)$$
For a hypothesis $h$,we define the training error (also called the empirical risk or empirical error in learning theory) to be
$$\hat{\epsilon}(h)=\frac 1 m \sum_{i=1}^m 1\{h(x^{(i)}) \ne y^{(i)}\}$$
$$\epsilon(h)=P_{(x,y) \sim D}(h(x) \ne y).$$
$$\hat{\epsilon} (h_i)= \frac 1 m \sum_{j=1}^m Z_j.$$
$$\epsilon(\hat{h}) \le \epsilon (h_*) + 2\gamma$$
Theorem. Let $|\mathcal{H}|=k$, and let any $m,\sigma$ be fixed. Then with probability at least $1-\sigma$, we have that
$$\epsilon(\hat{h}) \le \left ( \min_{h\in \mathcal{H}} \epsilon(h) \right )+ 2\sqrt{\frac 1 2m log \frac {2k} \sigma}$$
Corollary. Let $|\mathcal{H}|=k$, and let any $\,\gamma$ be fixed. Then for $\epsilon(\hat{h}) \le \min_{h \sim \mathcal{H}} \epsilon(h) + 2\gamma$ to hold with probability at least $1-\delta$, it suffices that
$$\begin{align} m & \ge \frac 1 {2\gamma^2} log \frac {2k} \delta \\ & = O\left(\frac 1 {\gamma^2}log \frac k \delta \right). \end{align}$$

Factor Analysis:

Marginals and conditionals of Gaussians,

$$\begin{align}Cov(x) & = \Sigma \\ & = \begin{bmatrix} \Sigma_{11} && \Sigma_{12} \\ \Sigma_{21} && \Sigma_{22} \end{bmatrix} \\ & = E[(x-\mu)(x-\mu)^T] \\ & =E\begin{bmatrix} \begin{pmatrix} x_1-\mu_1 \\ x_2-\mu_2 \end{pmatrix} & {\begin{pmatrix} x_1-\mu_1 \\ x_2-\mu_2 \end{pmatrix}}^T \end{bmatrix} \\ & =E\begin{bmatrix} (x_1-\mu_1){(x_1-\mu_1)}^T & (x_1-\mu_1){(x_2-\mu_2)}^T \\ (x_2-\mu_2){(x_1-\mu_1)}^T & (x_2-\mu_2){(x_2-\mu_2)}^T \end{bmatrix}\end{align}$$
$$\begin{align} \mu_{1|2} & = \mu_1+\Sigma_{12}\Sigma_{22}^{-1}(x_2-\mu_2) \\ \Sigma_{1|2} & = \Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21} \end{align}$$
To deduce the above marginals,we define $V \in R^{(m+n)\times (m+n)}$ need the lemma below:
$$\begin{align} V = \begin{bmatrix} V_{AA} & V_{AB}\\ V_{BA} & V_{BB} \end{bmatrix}=\Sigma^{-1} \\ \begin{bmatrix} A & B\\ C & D \end{bmatrix}^{-1}=\\ \begin{bmatrix} M^{-1} & -M^{-1}BD^{-1}\\ -D^{-1}C M^{-1} & D^{-1}+D^{-1}CM^{-1}BD^{-1} \end{bmatrix} \end{align}$$
where $M=A-BD^{-1}C.$ Using this formula, it follows that
$$\begin{align} \begin{bmatrix} \Sigma_{AA} & \Sigma_{AB}\\ \Sigma_{BA} & \Sigma_{BB} \end{bmatrix}&=\begin{bmatrix} V_{AA} & V_{AB}\\ V_{BA} & V_{BB} \end{bmatrix}^{-1}\\ &= \begin{bmatrix} (V_{AA}-V_{AB}V_{BB}^{-1}V_{BA})^{-1} & -(V_{AA}-V_{AB}V_{BB}^{-1}V_{BA})^{-1}V_{AB}V_{BB}^{-1}\\ -V_{BB}^{-1}V_{BA} (V_{AA}-V_{AB}V_{BB}^{-1}V_{BA})^{-1} & (V_{BB} - V_{BA}V_{AA}^{-1}V_{AB})^{-1} \end{bmatrix} \end{align}$$
And the “completion of squares” trick.Consider the quadratic function $z^TAz+b^Tz+c$ where $A$ is a symmetric,nonsingular matrix.Then, one can verify directly that
$$\frac 12 z^TAz+b^Tz+c=\frac 12 (z+A^{-1}b)^TA(z+A^{-1}b)+c-\frac 12 b^TA^{-1}b.$$
EM for factor analysis
In the factor analysis model, we posit a joint distribution on $(x,z)$ as follows,where $z \in \mathbb{R}^k$ is a latent random variable:
$$\begin{align} z & \sim \mathcal{N}(0,I)\\ x|z & \sim \mathcal{N}(\mu + \Lambda z,\Psi). \end{align}$$
Here, the parameters of our model are the vector $\mu \in \mathbb{R}^n$, the matrix $\Lambda \in \mathbb{R}^{n \times k}$. The value of $k$ is usually chosen to be smaller than $n$.
Thus, we imagine that each datapoint $x^{(i)}$ is generated by sampling a $k$ dimension multivariate Gaussian $z^{(i)}$. Then, it is mapped to a $k$-dimensional affine space of $\mathbb{R}^n$ by computing $\mu + \Lambda z^{(i)}$. Lastly, $x^{(i)}$ is generated by adding covariance $\Psi$ noise to $\mu + \Lambda z^{(i)}$.
$$\begin{align} z & \sim \mathcal{N}(0,I) \\ \epsilon & \sim \mathcal{N}(0,\Psi) \\ x & = \mu+\Lambda z+\epsilon \end{align}$$
where $\epsilon$ and $z$ are independent.
$$\begin{bmatrix} z \\ x \end{bmatrix} \sim \mathcal{N} \begin{pmatrix} \begin{bmatrix} \vec 0 \\ \mu \end{bmatrix} , & \begin{bmatrix} I & \Lambda^T \\ \Lambda & \Lambda \Lambda^T+\Psi \end{bmatrix} \end{pmatrix}$$
$$\begin{align} \ell(\mu,\Lambda,\Psi)=log \prod_{i=1}^m {1 \over (2\pi)^{n/2}\left\vert \Lambda\Lambda^T+\Psi \right\vert^{1/2}}exp\left( -\frac 12 (x^{(i)}-\mu)^T(\Lambda\Lambda^T+\Psi)^{-1}(x^{(i)}-\mu) \right). \end{align}$$
$$\begin{align} z^{(i)}|x^{(i)};\mu,\Lambda,\Psi \sim \mathcal{N}(\mu_{z^{(i)}|x^{(i)}},\Sigma_{z^{(i)}|x^{(i)}}) \\ \mu_{z^{(i)}|x^{(i)}} & = \Lambda^T(\Lambda \Lambda^T+ \Psi)^{-1}(x^{(i)}-\mu), \\ \Sigma_{z^{(i)}|x^{(i)}} & = I - \Lambda^T(\Lambda \Lambda^T+ \Psi)^{-1}\Lambda \\ \Lambda & = \left( \sum_{i=1}^m (x^{(i)}-\mu)\mu^T_{z^{(i)}|x^{(i)}} \right) \left( \sum_{i=1}^m \mu_{z^{(i)}|x^{(i)}} \mu^T_{z^{(i)}|x^{(i)}}+\Sigma_{z^{(i)}|x^{(i)}} \right)^{-1} \\ \mu & = \frac 1m \sum_{i=1}^m x^{(i)} \\ \Phi & = \frac 1m \sum_{i=1}^m x^{(i)}{x^{(i)}}^T-x^{(i)}\mu^T_{z^{(i)}|x^{(i)}}\Lambda^T-\Lambda\mu_{z^{(i)}|x^{(i)}}{x^{(i)}}^T \\ & \quad +\Lambda(\mu_{z^{(i)}|x^{(i)}}\mu^T_{z^{(i)}|x^{(i)}}+\Sigma_{z^{(i)}|x^{(i)}})\Lambda^T \end{align}$$
setting $\Psi_{ii}=\Phi_{ii}$ ( i.e. ,letting $\Psi$ be the diagonal matrix containing only the diagonal entries of $\Phi$).