Exercise 17.3 - EM for HMMs with mixture of Gaussian observations

The complete likelihood takes the form:

where $\pi$ is the initial distribution of the hidden state, $\mathbf{\text{𝐀}}$ is a $J\ast J$ matrix with transition probability, $\mathbf{\text{𝐖}}$ is a $J\ast K$ matrix for $w_{\mathit{𝑗𝑘}}$, $\mu$ and $\mathrm{\Sigma }$ are tensors whose $(j,k)$-th component denote ${\mu }_{\mathit{𝑗𝑘}}$ and ${\mathrm{\Sigma }}_{\mathit{𝑗𝑘}}$ respectively. Now its logarithm reads (we temporarily drop the condition on paramters for conciseness):

When being taken expectation w.r.t. $p({\mathbf{\text{𝐳}}}_{1:T}|{\mathbf{\text{𝐱}}}_{1:T})$, the only terms that matter are identical to those in exercise 17.1. This finishes the E-step for this model.

For the M-step, the update for $\mathbf{\text{𝐀}}$ and $\pi$ are identical to those for an ordinary HMM since their gradients remain the same. To update $\mathbf{\text{𝐖}},\mu ,\mathrm{\Sigma }$, note that their dependence on the auxiliary function is through:

This is tantamount to estimate the parameters for $J\ast K$ Gaussian components independently, with extra weight on each sample. Let us denote

so ${\alpha }_{n,t}(j)$ is determined from the old set of parameters, then consider the likelihood, in which we use a one-hot vector of length $J\ast K$ to embed the new latent variables:

Although tedious, one can consider $(n,t)$ as the complex index for data, $(j,k)$ as the complex index for Gaussian components. Now the internal auxiliary function reads (the evidence of the latent variables does not depend on the new parameters and is thus omitted):

where:

in which $p(z_{n,t}=j)={\alpha }_{n,t}(j)$ and $p(h_{n,t,j,k}=1|z_{n,t}=j)$ can be computed as in an ordinary GMM by focusing on the $K$ Gaussian components under the hidden state $j$. This concludes the E-step for the second auxiliary function. The M-step for the $Q^{\text{I}}$ is thus similar to that for an ordinary, except for the introduction of an extra factor. This also completes the rest M-step for the first auxiliary function.