Exercise 13.2 - Derivation of M-step for EB for linear regression

We give the EM for Automatic Relevance Determination(ARD) for the linear regression scene, the model is defined by:

In which the latent variables are $\mathbf{\text{𝐰}}$, and it is the hyperparameters $\mathbf{\text{𝐀}}$ that requires estimation.

During the E-step, we are to estimate the expectation of $\mathbf{\text{𝐰}}$, conditioned on $\mathbf{\text{𝐗}}$ and $\mathbf{\text{𝐘}}$. Recall (4.125):

where:

We are now ready to write down the logarithm of the complete likelihood:

Hence the auxiliary function is (let $𝜃=(\beta ,\mathbf{\text{𝐀}})$):

The dependence of $Q$ on $\mathbf{\text{𝐀}}$ is through:

Hence:

Using a conjugate onto ${\alpha }_j$ yields (13.166).

Finally, the dependence of $Q$ on $\beta$ is through:

where:

Hence we have:

Adding a Gamma prior results in (13.168).