Exercise 18.1 - Derivation of EM for LG-SSM

We directly work on the auxiliary function, in which $𝜃$ denotes the collection of all six matrices. The dependence on $\mathbf{\text{𝐮}}$ is omitted for simplicity. The first subscript for variables $\mathbf{\text{𝐳}}$ and $\mathbf{\text{𝐲}}$ is the index of the sequence. :

For the E-step, we are to compute:

respectively, in which $\mathbf{\text{𝐄}}$ can be ${\mathrm{\Sigma }}_0^{-1}$, ${\mathbf{\text{𝐐}}}_i^{-1}$, ${\mathbf{\text{𝐀}}}_i^{\text{T}}{\mathbf{\text{𝐐}}}_i^{-1}{\mathbf{\text{𝐀}}}_i$ and ${\mathbf{\text{𝐂}}}_i^{\text{T}}{\mathbf{\text{𝐑}}}_i^{-1}{\mathbf{\text{𝐂}}}_i$, while $\mathbf{\text{𝐅}}$ is ${\mathbf{\text{𝐐}}}_i^{-1}{\mathbf{\text{𝐀}}}_i$. We then evoke the linearity of the trace operator:

Therefore we only need the conditional expectation of ${\mathbf{\text{𝐳}}}_{n,i}$, ${\mathbf{\text{𝐳}}}_{n,i}{\mathbf{\text{𝐳}}}_{n,i}^{\text{T}}$ and ${\mathbf{\text{𝐳}}}_{n,i}{\mathbf{\text{𝐳}}}_{n,i-1}^{\text{T}}$. The first two variables have been covered by (18.56)-(18.59). For the last variable, we simply need to borrow the form from (18.63). Adding all these together completes the E-step.

For the M-step, we should note that the auxiliary function after the E-step has become the form such that for ${\mu }_0$:

whose form is close to that for the mean in an ordinary MVN. For ${\mathrm{\Sigma }}_0$, ${\mathbf{\text{𝐐}}}_i$ and ${\mathbf{\text{𝐑}}}_i$, the conclusion is similar. As for matrices such as ${\mathbf{\text{𝐀}}}_i$, ${\mathbf{\text{𝐁}}}_i$, ${\mathbf{\text{𝐂}}}_i$ and ${\mathbf{\text{𝐃}}}_i$, note that $\frac{\mathit{\partial 𝑄}}{\partial {\mathbf{\text{𝐀}}}_i}$ is composed of a series with form:

and

(assuming symmetry!) Hence we end up with an involving update formula between ${\mathbf{\text{𝐀}}}_i$ and ${\mathbf{\text{𝐁}}}_i$, so is that for ${\mathbf{\text{𝐂}}}_i$ and ${\mathbf{\text{𝐃}}}_i$. This finishes the M-step for the EM for LG-SSM.