Question
Exponential Family. Verify that the multivariate Gaussian distribution can be casted in exponential family form and derive expressions for η , b(η) , T(η) , a(η) .
Solution
The exponential family form is
p(y;η)=b(y)exp[ηTT(y)−a(η)]
The multivariate Gaussian distribution takes the form
N(x|μ,Σ)=1(2π)D/2|Σ|1/2exp[−12(x−μ)TΣ−1(x−μ)]
Since
−12(x−μ)TΣ−1(x−μ)=−12(xT−μT)Σ−1(x−μ)=−12(xTΣ−1x−xTΣ−1μ−μTΣ−1x+μTΣ−1μ)=−12(xTΣ−1x−2μTΣ−1x+μTΣ−1μ)=−12(vec(Σ−1)vec(xxT)−2μTΣ−1x+μTΣ−1μ)=−12([vec(Σ−1)−2μTΣ−1][vec(xxT)x]+μTΣ−1μ)=[−12vec(Σ−1)μTΣ−1][vec(xxT)x]−12μTΣ−1μ
The 3rd equation is due to
( Trace of a real number is still a real number and Σ−1 is a symmetric matrix)
xTΣ−1μ=Tr(xTΣ−1μ)=Tr(xTΣ−1μ)T=Tr(μTΣ−Tx)=Tr(μTΣ−1x)=μTΣ−1x
The 4th equation is due to xTΣ−1x=vec(Σ−1)Tvec(xxT) , where vec means stack all columns of a matrix into one single column, i.e. if x=⎡⎣⎢|x1||x2||…||xm|⎤⎦⎥ , then
vec(x)=⎡⎣⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢|x1|x2|⋮|xm|⎤⎦⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥
So we can derive the parameter of exponential family as below:
b(y)=1(2π)D/2η=⎡⎣−12vec(Σ−1)μTΣ−1⎤⎦T(y)=[vec(xxT)x]a(η)=12μTΣ−1μ−ln|Σ|1/2