Study notes for Latent Dirichlet Allocation

1. Topic Models

• Topic models are based upon the idea that documents are mixtures of topics, where a topic is a probabilistic distribution over words. A topic model is a generative model for documents: it specifies a simple probabilistic procedure by which documents can be generated (Steyvers and Griffiths, 2007). Two general steps are taken to make a new document: 
  • Step 1, for each document, one chooses a distribution over topics.
  • Step 2, to generate each word in that document, one chooses a topic at random according to the distribution (because one same word may belong to various topics with different probabilities.). Then a word is drawn from the chosen topic in terms of probabilistic sampling, e.g. as illustrated in Figures 1 and 2. 

• When fitting a generative model, the goal is to find the best set of latent variables that can explain the observed data (i.e., observed words in documents), assuming that the model actually generated the data. 
• Many different generative models have been proposed under the same assumption that a document is a mixture of topics, but make slightly different statistical assumptions. 
• The number of topics will affect the interpretability of the results. A solution with too few topics will generally result in very broad topics whereas a solution with too many topics will result in uninterpretable topics that pick out idiosyncratic word combinations. One way is to choose the number of topics that leads to best generalization performance to new tasks. 
• Notations:
  • P(z) is the probability distribution over topics z in a particular document
  • P(w|z) is the probability distribution over words w given topic z. 
  • The model specifies the distribution over words within a document is:
    
    where t is the number of topics, is the probability that the j-th topic is chosen/sampled for the i-th word token, and