Exercise 24.2 - Gibbs sampling for a 1D Gaussian mixture model

Answers

The likelihood for this model is (assume that there are $K$ Gaussian components, we discuss this problem for MVN):

\begin{aligned} p (𝐗, 𝐙, π, μ, Σ) & = p (π) \cdot (\prod_{k = 1}^{K} p (μ_{k}) \cdot p (Σ_{k})) \cdot (\prod_{n = 1}^{N} p (𝐳_{n} | π) \cdot p (𝐱_{n} | 𝐳_{n}, μ, Σ)) \\ = \frac{1}{B (α)} \cdot \prod_{k = 1}^{K} π_{k}^{α_{k} - 1} \\ \cdot \prod_{k = 1}^{K} 𝒩 (μ_{k} | 𝐦_{0}, 𝐕_{0}) IW (Σ_{k} | 𝐒_{0}, v_{0}) \\ \cdot \prod_{n = 1}^{N} \prod_{k = 1}^{K} {(π_{k} 𝒩 (𝐱_{n} | μ_{k}, Σ_{k}))}^{z_{𝑛𝑘}} . \end{aligned}

For Gibbs sampling, we derive the conditional distribution of $𝐙$ , $π$ , $μ$ and $Σ$ conditioned on all other variables.

For the latent variables:

\begin{aligned} p (z_{𝑛𝑘} = 1 | 𝐗, π, μ, Σ) & = \frac{p (z_{𝑛𝑘} = 1, 𝐗, π, μ, Σ)}{p (𝐗, π, μ, Σ)} \\ \propto p (z_{𝑛𝑘} = 1 | π) \cdot p (𝐱_{n} | z_{𝑛𝑘} = 1, μ, Σ) \\ = π_{k} \cdot 𝒩 (𝐱_{n} | μ_{k}, Σ_{k}) . \end{aligned}

For the weights:

\begin{aligned} p (π | 𝐗, 𝐙, π, μ, Σ) & = \frac{p (π, 𝐗, 𝐙, π, μ, Σ)}{p (𝐗, 𝐙, π, μ, Σ)} \\ \propto p (π) \cdot p (𝐙 | π) \\ \propto (\prod_{k = 1}^{K} π_{k}^{α_{k} - 1}) \cdot (\prod_{n = 1}^{N} \prod_{k = 1}^{K} π_{k}^{z_{𝑛𝑘}}) \\ = \prod_{k = 1}^{K} π_{k}^{α_{k} + (\sum_{n = 1}^{N} z_{𝑛𝑘}) - 1} . \end{aligned}

Hence the conditional distribution for $π$ follows (24.11).

For the means:

\begin{aligned} p (μ_{k} | 𝐗, 𝐙, π, μ_{- k}, Σ) & = \frac{p (μ_{k}, 𝐗, 𝐙, π, μ_{- k} Σ)}{p (𝐗, 𝐙, π, μ_{- k} Σ)} \\ \propto p (μ_{k}) \cdot p (𝐗 | μ_{k}, 𝐙, Σ, μ_{- k}) \\ = 𝒩 (μ_{k} | 𝐦_{0}, 𝐕_{0}) \cdot \prod_{n = 1}^{N} {(π_{k} 𝒩 (𝐱_{n} | μ_{k}, Σ_{k}))}^{z_{𝑛𝑘}} . \end{aligned}

At this point it is better to observe in the logarithm perspective:

\begin{aligned} \log p (μ_{k} | 𝐗, 𝐙, π, μ_{- k}, Σ) & = - \frac{1}{2} {(μ_{k} - 𝐦_{0})}^{T} 𝐕_{0}^{- 1} (μ_{k} - 𝐦_{0}) \\ + \sum_{n = 1}^{N} - \frac{z_{𝑛𝑘}}{2} {(μ_{k} - 𝐱_{n})}^{T} Σ_{k}^{- 1} (μ_{k} - 𝐱_{n}) + const, \end{aligned}

from which we observe that the coefficient for $μ^{T} μ$ and $μ$ are:

- \frac{1}{2} (𝐕_{0}^{- 1} + Σ_{k}^{- 1} \sum_{n = 1}^{N} z_{𝑛𝑘}),

𝐕_{0}^{- 1} 𝐦_{0} + \sum_{n = 1}^{N} z_{𝑛𝑘} Σ_{k}^{- 1} 𝐱_{n} .

Therefore the conditional distribution for $μ_{k}$ is a Gaussian with covariance:

𝐕_{k} = {(𝐕_{0}^{- 1} + Σ_{k}^{- 1} \sum_{n = 1}^{N} z_{𝑛𝑘})}^{- 1}

and mean:

𝐦_{k} = 𝐕_{k} (𝐕_{0}^{- 1} 𝐦_{0} + \sum_{n = 1}^{N} z_{𝑛𝑘} Σ_{k}^{- 1} 𝐱_{n}) .

Finally, for the covariance:

\begin{aligned} p (Σ_{k} | 𝐗, 𝐙, π, μ, Σ_{- k}) & = \frac{p (Σ_{k}, 𝐗, 𝐙, π, μ, Σ_{- k})}{p (𝐗, 𝐙, π, μ, Σ_{- k})} \\ \propto p (Σ_{k}) \cdot p (𝐗 | Σ_{k}, 𝐙, μ, Σ_{- k}) . \end{aligned}

This is tantamount to:

\begin{aligned} \log p (Σ_{k} | 𝐗, 𝐙, π, μ, Σ_{- k}) & = - \frac{v_{0}}{2} \cdot \log | Σ_{k} | - \frac{1}{2} tr [𝐒_{0}^{- 1} Σ_{k}^{- 1}] \\ + \sum_{n = 1}^{N} z_{𝑛𝑘} (- \frac{1}{2} \cdot \log | Σ_{k} | - \frac{1}{2} {(𝐱_{n} - μ_{k})}^{T} Σ_{k}^{- 1} (𝐱_{n} - μ_{k})) + const \\ = - \frac{v_{0} + \sum_{n = 1}^{N} z_{𝑛𝑘}}{2} \cdot \log | Σ_{k} | \\ - \frac{1}{2} tr [(𝐒_{0}^{- 1} + \sum_{n = 1}^{N} z_{𝑛𝑘} {(𝐱_{n} - μ_{k})}^{T} (𝐱_{n} - μ_{k})) Σ_{k}^{- 1}] + const . \end{aligned}

Hence the conditional distribution of $Σ_{k}$ is the inverse-Wishart distribution with:

𝐯_{k} = v_{0} + \sum_{n = 1}^{N} z_{𝑛𝑘},

𝐒_{k} = {(𝐒_{0}^{- 1} + \sum_{n = 1}^{N} z_{𝑛𝑘} {(𝐱_{n} - μ_{k})}^{T} (𝐱_{n} - μ_{k}))}^{- 1} .

The difference from (24.18) is in the definition of $𝐒_{0}$ .

solour_lfq

2021-03-24 13:42