A Note on Kaldi's PLDA Implementation

Kaldi’s PLDA implementation is based on [1], the so-called two-covariance PLDA by [2]. The authors derive a clean update formula for the EM training and give a detailed comment in the source code. Here we add some explanations to make formula derivation more easy to catch.

A pdf version of this note can be found here

1. Background

Recall that PLDA assume a two stage generative process:
1) generate the class center according to

$$y \sim \mathcal{N}(\mu, \Phi_b)$$
2) then, generate the observed data by:
$$x \sim \mathcal{N}(y, \Phi_w)$$

Here,$\mu$ is estimated by the global mean value:

$$\mu = \sum_{k=1}^K \sum_{i=1}^{n_k} z_{ki}$$
here $z_{ki}$ depicts the $i$-th sample of the $k$-th class.

So let’s to the estimation of $\Phi_b$ and $\Phi_w$.

Note that, as $\mu$ is fixed, we remove it from all samples. Hereafter, we assume all samples have pre-processed by removing $mu$ from them.

The prior distribution of an arbitrary sample $z$ is:

$$p(z) \sim \mathcal{N}(0, \Phi_w + \Phi_w)$$
Let’s suppose the mean of a particular class is $m$, and suppose that that class had $n$ examples.

$$m = \frac{1}{n}\sum_{i=1}^n z_i \sim \mathcal{N}(0, \Phi_w + \frac{\Phi_w}{n} )$$

i.e. $m$ is Gaussian-distributed with zero mean and variance equal to the between-class variance plus $1/n$ times the within-class variance. Now, $m$ is observed (average of all observed samples).

2. EM

We’re doing an E-M procedure where we treat $m$ as the sum of two variables:

$$m = x + y$$

where $x \sim N(0, \Phi_b)$, $y \sim N(0, \Phi_w/n)$.

The distribution of $x$ will contribute to the stats of Φb