Exercise 21.9 - Variational EM for binary FA with sigmoid link

We begin with the likelihood:

The prior for the hidden variables is assumed to be:

Assume the factorized variational distribution:

For the variational E-step, our goal is to match the logarithm of the variational distribution on the hidden variables:

with:

We can see that this form cannot painlessly reduce to an exponential family, hence approximation needs to be conducted to transfer $\log <!--\; FUNCTION\; APPLICATION\; -->(1--\text{sigm}({\mathbf{\text{𝐰}}}_j^{\text{T}}{\mathbf{\text{𝐳}}}_n))$ to a linear function of ${\mathbf{\text{𝐳}}}_n$ and optinally ${\mathbf{\text{𝐳}}}_n^{\text{T}}{\mathbf{\text{𝐳}}}_n$ (e.g., the Laplace approximation). Then we can see that ${𝔼}_{q(\mathbf{\text{𝐖}})}\left[\log <!--\; FUNCTION\; APPLICATION\; -->p(\mathbf{\text{𝐙}},\mathbf{\text{𝐗}},\mathbf{\text{𝐖}})\right]$ is a quadratic function in ${\mathbf{\text{𝐳}}}_n$, hence the E-step reduces $q(\mathbf{\text{𝐳}})$ to a Gaussian.

For the variational M-step:

can again be approximated as a quadratic function w.r.t. $\mathbf{\text{𝐖}}$, where the expectation of $\mathbf{\text{𝐳}}$ shall be replaced by their counterpart in the E-step.