Exercise 11.11 - Visible mixtures of Gaussians are in exponential family

Answers

Encode latent variable as binary code:

z_{k} = 𝕀 (x is generated from the k -th base distribution),

then

\begin{aligned} p (𝐳 | 𝜃) & = \prod_{k = 1}^{K} π_{k}^{z_{k}}, \\ p (x | 𝐳, 𝜃) & = \prod_{k = 1}^{K} {(\frac{1}{\sqrt{2 π σ_{k}^{2}}} \cdot \exp {- \frac{1}{2 σ_{k}^{2}} {(x - μ_{k})}^{2}})}^{z_{k}}, \end{aligned}

where $𝜃 = (π, μ, σ^{2})$ .

The logarithm for the joint distribution is:

\begin{align} \log p (x, 𝐳 | 𝜃) = & \log \prod_{k = 1}^{K} {(\frac{π_{k}}{\sqrt{2 π σ_{k}^{2}}} \cdot \exp {- \frac{1}{2 σ_{k}^{2}} {(x - μ_{k})}^{2}})}^{z_{k}} \\ = & \sum_{k = 1}^{K} z_{k} \cdot (\log π_{c} - \frac{1}{2} \log 2 π σ_{k}^{2} - \frac{1}{2 σ_{k}^{2}} {(x - μ_{k})}^{2}) . \end{align}

To reduce $p (x, 𝐳 | 𝜃)$ into the exponential family, note that $\log p (x, 𝐳 | 𝜃)$ is linearly dependent on $𝐳$ and $x$ , hence we can rewrite:

ϕ (x, 𝐳) = {(𝐳^{T}, x 𝐳^{T}, x^{2} 𝐳^{T})}^{T},

the parameters for this form are:

{(\log π_{\dots} - \frac{1}{2} \log 2 π σ^{2}, μ_{\dots} ⊙ σ_{\dots}^{- 2}, - \frac{1}{2 σ_{\dots}^{2}})}^{T}

in vector form.

For the mixture of MVN, since:

\log p (𝐱, 𝐳 | 𝜃) = \sum_{k = 1}^{K} z_{k} {(\log π_{c} - \frac{D}{2} \log 2 π - \frac{1}{2} \log | Σ_{k} | - \frac{1}{2} {(𝐱 - μ_{k})}^{T} Σ_{k}^{- 1} (𝐱 - μ_{k}))}^{2},

so the distribution is still an exponential family member, with sufficient statistics:

ϕ (𝐱, 𝐳) = (𝐳, 𝐱 ⊙ 𝐳, 𝐱 ⊙ 𝐱 ⊙ 𝐳),

rearranged as a vector. ( $⊙$ denotes the tensor/outer product.)

solour_lfq

2021-03-24 13:42