I. Notations
X={x1,x2,...,xN} i.i.d. observed variables
Z={z1,z2,...,zN} latent variables
Θ(t) The estimate of the parameters at iteration t
II. Derivations
1. Maximum-Likelihood:
which is hard to compute with a gradient method.
where q(z | x,Θ) is an arbitrary density over Z , and the inequality is given by Jessen’s inequality, i.e,
2. Expectation-Maximization:
Thus we have the lower bound of target function
l(Θ)
. Instead of maximizing
l(Θ)
directly, EM maximizes the lower-bound
Q(q,Θ)
via coordinate ascent:
E-Step: compute
q(t+1)=argmaxqQ(q,Θt)
with constraint
∑Kkq(z=k | x,Θ)=1
(arbitrary density function over
Z
), by introducing the lagrange multiplier
thus, q=P(z|x,Θ) give the closest lower bound of l(Θ)
M-Step: update parameters
θ
, with
III. Applications to Gaussian Mixture Models(GMMs)
For general mixture models, we have
For Gaussian Mixture Models (GMMs), we have
E-Step for GMMs:
Define
qi,k≡P(z=k|xi,Θ)
, then
M-Step for GMMs:
update the parameters which maximizes the Log-likelihood below
with subject to ∑Kkπk=1 , introducing the lanrange multiplier into the objective function, we thus have