Dimensinality reduction and topic modeling

Dimensinality reduction and topic modeling

0 Abstract

BOW is not effective facing synonym and polysemy.
Dimension reduction can solve this, represent a document in lower-dimension, and reflect concepts.

Two froms of dimension reduction:

• latent semantic indexing using spectral decomposition
• Topic modeling PLSI & LDA using probabilistic model to find the co-occurrence patterns which correspond to semantic topic

Then a survey of advances to apply these techiniques to large and envolving datasets and to in corporate network and contextual information

1 Introduction

• Index of Hebrew difficulties
  • suppress difference sin which were not significant.
  • preserving diffrences that might affect the sematics.
• core task automated text-mining
  • synonymy & ploysemy
• Bag of words (BOW)
  • acounting for frequency ignoring order
  • high dimensionality and sparse (term-document matrix)
• relationship between Clustering, Reduction and Topic modeling
  • (Discriminate method) Clustering & soft clustering: uses similarity, natural cluster but hard to interpretate. soft (associate with multiple clusters)
  • (Discriminate method) Dimension reduction: BOW representation, more original and coupling, still hard to interpretate.
  • (Generative method) Topic modeling: combination of soft clustering and dimension reduction.

2 latnet semantic indexing

projecting documents into semantic space, analysising at conceptual level.

• Overcome synonymy and ploysemy (help term-based information retrieval)

• From 1980s

  • information retrieval
  • assigning papers to reviewers
  • cross-lingual retrieval

• Based on the approximation SVD of the term-document matrix, view low-rank space as semantic concepts space

• Procedure

  • SVD
    $$X=U\Sigma V^T$$
  • low-rank approximation (minimizes spectral norm and Frobenius norm).
    $$\begin{array}{rl}\hat{X}& =\hat{U}\hat{\Sigma }{\hat{V}}^T=\left[\begin{array}{lll}{\mathbf{U}}_1& \dots & {\mathbf{U}}_K\end{array}\right]\left[\begin{array}{lllc}{\sigma }_1& & & \\ & \ddots & & \\ & & & {\sigma }_K\end{array}\right]\left[\begin{array}{c}{\mathbf{V}}_1^T\\ ⋮\\ {\mathbf{V}}_K^T\end{array}\right]\end{array}$$
  • document and term representations
    $${\mathbf{X}}_d=\hat{U}\hat{\Sigma }{\hat{\mathbf{X}}}_d{\mathbf{T}}_v=\hat{V}\hat{\Sigma }{\hat{\mathbf{T}}}_v$$
  • application
    • Information retrieval
      $$\hat{\mathbf{q}}={\hat{\Sigma }}^{-1}{\hat{U}}^T\mathbf{q}$$
    • Document similarity: One solved non-identifiability of the SVD [63]
    • Term similartiy

• Implementation

  • Term-Document Matrix
    • DF*IDF
    • language pyramid model [70]
  • Computation
    • Lanczos algorithm (for sparse matrix)
  • Handling changes: corpus changes
    • fold in: computing new ones using original decomposition. Efficient O(KN).
    • Updating semantic space: [ 8 (1995) , 52 (1994) , 74 (1999) ]
      • [74]: $\left[\hat{X}{X}^{\prime }\right]$ instead of $\left[X{X}^{\prime }\right]$

• Some Analysis