Exercise 7.9 - Generative model for linear regression

For question (a), assume that $\mathbf{\text{𝐱}}$ and $y$ are jointly subject to a Gaussian:

We know from (4.68) that the marginal distribution of $\mathbf{\text{𝐱}}$ and $y$ are $𝒩(\mathbf{\text{𝐱}}|{\mu }_{\mathbf{\text{𝐱}}},{\mathrm{\Sigma }}_{\mathit{𝑥𝑥}})$ and $𝒩(\mathbf{\text{𝐲}}|{\mu }_y,{\mathrm{\Sigma }}_{\mathit{𝑦𝑦}})$ respectively, this is sufficient for MLE ${\mu }_{\mathbf{\text{𝐱}}}$, ${\mu }_y$, ${\mathrm{\Sigma }}_{\mathit{𝑥𝑥}}$ and ${\mathrm{\Sigma }}_{\mathit{𝑦𝑦}}$:

For an estimation of ${\mathrm{\Sigma }}_{\mathit{𝑥𝑦}}$, connecting $\mathbf{\text{𝐱}}$ and $y$ together then estimating this vector’s covariance matrix (this is an ordinary Gaussian model), picking only the first $D$ components from its last column:

Finally, plugging our observations into (4.69):

which equals:

This is identical to what (7.109)-(7.111) implies, hence the proof is completed.

For question (b), there is no significant difference except that we are now equipped with a distribution of $x$. This enables us the ability of generating more samples and sheds light on active querying strategies, in which we can query an oracle for samples in order to accelerate the convergence of the learning process.