Exercise 11.6 - EM for a finite scale mixture of Gaussians

Answers

For question (a), we advance straightforwardly:

\begin{aligned} p (J_{n} = j, K_{n} = k | x_{n}, 𝜃) & = \frac{p (x_{n}, J_{n} = j, K_{n} = k | 𝜃)}{p (x_{n}, 𝜃)} \\ = \frac{1}{A} p_{j} \cdot q_{k} \cdot 𝒩 (x_{n} | μ_{j}, σ_{k}^{2}) \\ = \frac{p_{j} \cdot q_{k} \cdot 𝒩 (x_{n} | μ_{j}, σ_{k}^{2})}{\sum_{j^{'}, k^{'}} p_{j^{'}} \cdot q_{k^{'}} \cdot 𝒩 (x_{n} | μ_{j^{'}}, σ_{k^{'}}^{2})} . \end{aligned}

For question (b):

\begin{aligned} Q (𝜃^{new}, 𝜃^{old}) & = 𝔼_{p (J_{n}, K_{n} | x_{n}, 𝜃^{old})} [\sum_{n = 1}^{N} \log p (x_{n}, J_{n}, K_{k} | x_{n}, 𝜃^{new})] \\ = \sum_{n = 1}^{N} 𝔼_{p (J_{n}, K_{n} | x_{n}, 𝜃^{old})} [\sum_{j = 1}^{m} \sum_{k = 1}^{l} z_{𝑗𝑘} \log (p_{j}^{new} q_{k}^{new} \cdot 𝒩 (x_{n} | μ_{j}^{new}, σ_{k}^{2, new}))] \\ = \sum_{n} \sum_{j, k} 𝔼_{p (J_{n}, K_{n} | x_{n}, 𝜃^{old})} [z_{𝑗𝑘}] (\log p_{j}^{new} + \log q_{k}^{new} + \log 𝒩 (x_{n} | μ_{j}^{new}, σ_{k}^{2, new})) . \end{aligned}

What left is trivial calculus. Define:

r_{𝑛𝑗𝑘} = 𝔼_{p (J_{n}, K_{n} | x_{n}, 𝜃^{old})} [z_{𝑗𝑘}] = p (J_{n} = j, K_{n} = k | x_{n}, 𝜃)

For question (c), we have:

\begin{aligned} \frac{∂𝑄}{\partial μ_{j^{'}}^{new}} & = \sum_{n} \sum_{j, k} r_{𝑛𝑗𝑘} \cdot \frac{\partial}{\partial μ_{j^{'}}^{new}} \log 𝒩 (x_{n} | μ_{j}^{new}, σ_{j}^{2, new}) \\ = \sum_{n} \sum_{k} r_{n j^{'} k} \cdot \frac{(x_{n} - μ_{j^{'}}^{new})}{σ_{k}^{2, new}} . \end{aligned}

Setting it to zero yields:

μ_{j^{'}}^{new} = \frac{\sum_{n} \sum_{k} \frac{r_{n j^{'} k} \cdot x_{n}}{σ_{k}^{2, new}}}{\sum_{n} \sum_{k} \frac{r_{n j^{'} k}}{σ_{k}^{2, new}}} .

solour_lfq

2021-03-24 13:42