<Probabilistic Graphical Models>2 Graphical Models in Action

• From course Probabilistic Models and Inference Algorithms for Machine Learning, Prof. Dahua Lin
• All contents here are from the course and self understandings.

---

基本步骤

1. 理解你要求解的某个问题
   
   • what kind of entities/factors are involved?
   • How do they interact with each other?
   • Any constraints to take into account?
2. 建立模型
   
   • Introduce variables
   • Specify relations among them: make assumptions and modeling choices
   • Formalize the graphical model
3. Derive the inference & estimation algorithms

---

Gaussian Mixture Model (GMM)

1. Motivation & Assumptions

• Observation: clusters （如图中所示）
• Assumptions:
  
  • latent components
  • independent generation of components （一个component指一个如图所示的cluster）
  • independent generation of points in each component.

2. Formulate Gaussian Mixture Model

• Variables:
  
  • sample points: $x_i$
  • component indicators: $z_i$
• Generative procedure: for each i
  
  • choose a component: $z_i \in \pi$
  • generate a point: $x_i \in \mathcal N(\mu_{z_i}, \Sigma_{z_i})$ (每个sample point 服从正态分布，mean和方差为$\mu_{z_i}$ 和 $\Sigma_{z_i}$)
• Model parameters:
  
  • component parameters: $\{(\mu_k, \Sigma_k)\}_{k=1:K}$
  • choice prior: $\pi = (\pi_2 , \cdots , \pi_K)$
• Joint distribution:
  $$p(X,Z|\Theta) = \prod^N_{i=1} p_{\mathcal C}(z_i|\pi)p_{\mathcal N}(x_i|\mu_{z_i}, \Sigma_{z_i})$$
• 该模型的图形表达如下：
  
  
  • 但是以上的模型不能泛化：当有多个groups 的数据时候，每一个 group $G_m$ 都有一个 prior: $\pi_m$，这样每一个group一个先验，对于新来的数据，并没有一个generalized的 $\pi_{new}$. 因此需要一个 Group-wise 的GMM

3. Extend Gaussian Mixture Model

• 一般来说，要泛华某个先验（prior），我们可以采用一些常见的分布，如下，这里我们采用Dirichlet
  
• Group-wise GMM: Generalizable to New Groups
  
  • Introduce a Dirichlet Prior over $π_m$ to allow the generation of new groups.
  • Formulation:
  • • For each group $G_m: \pi_m ∼ Dir(α)$.
  • • Generate the i-th point in $G_m$: $$z_i ∼ \pi_m$$ $$x_i ∼ \mathcal N(\mu_k, \Sigma_k)$$
  • • 注意 $\pi_m$ 现在是个 隐变量 (latent variable)
      
• Temporal Structures:
  
  • 在实际世界中，时域上的变化是很常见的，那对于模型来说，也需要对应的dynamics作用到 不同的变量上
  • Three ways to incorporate dynamics：
    Dynamics on $x_i$
    Dynamics on $z_i$
    Dynamics on $\pi$