Week3-4Dimensionality reduction

Problems with the simple vector approaches to similarity

Dimensionality reduction

• looking for hidden similarities in data
• based on matrix decomposition

Matrix decomposition

SVD

Example

• Assume that we have 7 Documents with 9 terms.
  
  e.g. Document 1 contains term 6 and term9.

• The document term matrix should be $\mathbb{R}^{9 \times 7}$, a column represents a document and a raw represents a term.
  

Remark: we have to normalize our matrix before svd.

• Apply the svd decomposition
  

  $$M^{9 \times 7} = U^{9 \times 9} \Sigma^{9 \times 7} V^T$$

  

• $\Sigma$
  

• Rank 2 $\Sigma$
  

  $U\Sigma_2$ is the 2 rank approximation of the TERM(2 dimension),
  $\Sigma_2V^T$ is the 2 rank approximation of the DOCUMENT(2 dimension).

Question

what do $A^TA$ and $AA^T$ mean if A is a document-term matrix $\mathbb{R}^{9 \times 7}$?

• $A^TA \in \mathbb{R}^{7 \times 7}$ is the document-document similarity matrix.
• $AA^T \in \mathbb{R}^{9 \times 9}$ is the term-term similarity matrix.

Latent semantic indexing(LSI, identical to LSA)

• Dimensionality reduction = identification of hidden(latent) concepts
• query matching in latent space