Low-rank approximations

Low-rank approximations

Give $M \times N$ matrix $C$ and a positive integer $k$, we wish to find an $M \times N$ matrix $C_k$ of rank at most $k$，so as to minimize the Frobenius norm of the matrix difference $X=C-C_k$ ，defined to be

$$||X||_F = \sqrt{\sum_{i=1}^M\sum_{j=1}^N x_{ij}^2} \text{ .} \tag{1}$$

Thus, the Frobenius norm of $X$ measures the discrepancy between $C_k$ and $C$; our goal is to find a matrix $C_k$ that minimizes this discrepancy, while constraining $C_k$ to have rank at most $k$. If $r$ is the rank of $C$, clearly $C_r=C$ and the Frobenius norm of the discrepancy is zero in this case. When $k$ is far smaller than $r$, we refer to $C_k$ as a low-rank approximation.

---

SVD

1. Given $C$, constuct its SVD int the form of $C=U \Sigma V^T$.
2. Derive from $\Sigma$ the matrix $\Sigma_k$ formed by replacing by zeros the $r-k$ smallest sigular values on the diagnal of $\Sigma$.
3. Compute and output $C_k = U\Sigma_k V^T$ as the rank-$k$ approximation to $C$.

http://nlp.stanford.edu/IR-book/html/htmledition/low-rank-approximations-1.html