Exercise 21.10 - VB for binary FA with probit link

To attack this question, we assume the following variational distribution:

Since we have three independent family of variables to be estimated, this pseudo VB proceduce consists of three steps (not simple an E-step and an M-step). We assume $L=1$ w.l.o.g.

For the first step, we update for ${\mathbf{\text{𝐱}}}_i$, this is done by collecting terms relevent to ${\mathbf{\text{𝐱}}}_i$ from the expectation of the log likelihood w.r.t. $q(\mathbf{\text{𝐰}},\mathbf{\text{𝐙}})$:

From which we observe that $q({\mathbf{\text{𝐱}}}_i)$ takes the form of a Gaussian, the update for its precision is:

For its mean:

In the second step, we update $\mathbf{\text{𝐰}}$, note that:

Therefore the variational distribution for $\mathbf{\text{𝐰}}$ should better be a Gaussian, whose update takes the form:

Finally, we update for $z_i$:

moreover, the domain for $z_i$ is confined in $(0,\infty )$ if $y_i=1$ and $(-\infty ,0]$ otherwise. Therefore the variational distribution for $z_i$ is truncated, yet it has a quadratic form w.r.t. $z_i$ in its exponential, hence it is a truncated Gaussian as what has to be proven. The variance for this distribution (if not truncated) is uniformly unity, and its mean is:

All three update steps form a fixed-point fashion learning.