Exercise 12.1 - M-step for FA

The EM for FA, as a useful exercise if you want to become proficient at the math, is presented in detail as follows. As for the mixture of FAs, you can refer to The EM Algorithm for Mixtures of Factor Analyzers, Zoubin Gharamani, Geoffrey E.Hinton, 1996.

We begin with: (centralize $\mathbf{\text{𝐗}}$ to cancel $\mu$ w.l.o.g):

where we have centralized $\mathbf{\text{𝐗}}$ to simplify the deduction. Now we apply (4.124) and (4.125) to the two equations before, this ends up with:

The log-likelihood for the complete data set $\left\{\mathbf{\text{𝐱}},\mathbf{\text{𝐳}}\right\}$ is:

We are now ready to formulate the auxiliary function, let $𝜃=(\mathbf{\text{𝐖}},\mathrm{\Psi })$:

where the conditional first and second moments are:

Therefore:

Finally, let us take partial gradient of the auxiliary function w.r.t. $\mathbf{\text{𝐖}}$ and $\mathrm{\Psi }$. We start with $\mathbf{\text{𝐖}}$, note that $\mathrm{\Sigma }$ and ${\mathbf{\text{𝐦}}}_n$ only dependents on ${\mathbf{\text{𝐖}}}^{\text{old}}$, hence the first term is a constant for $𝜃$:

Setting it to zero yields:

Meanwhile, let $\mathrm{\Lambda }={\mathrm{\Psi }}^{-1}$:

Hence:

This completes the proof.