Exercise 7.8 - Bayesian linear regression in 1d with known $\sigma^{2}$

For question (a), the estimation for ${\sigma }^2$ is $0.3173$.

For question (b), we have:

To simplify the algebra, we observe that $w_1$ and $w_0$ are independent in this prior. Thus the prior distribution can be reduced to: $𝒩(w_1|0,1)\cdot 𝒩(w_0|\gamma ,\infty ),$ where $\gamma$ can be an arbitrary finite number, thus:

For question (c) and (d), we consider the posterior distribution for parameters:

where we only maintain terms dependent on $w_0$ and $w_1$. To marginalize out $w_0$, we have:

Hence the posterior distribution over $w_1$ is a normal distribution. The coefficients for $w_1^2$ and $w_1$ in the exponential are respectively:

Thence its posterior variance is:

its mean is:

Finally, let us plug $A$ to $E$ with statistics of $𝒟$:

The posterior variance is:

from which we observe that, with $N$ grows, the denominator increases as the bound from the Cauchy inequality:

To put in other words, The larger the Cauchy difference is, the more confidence we have for the estimation of $w_1$. Such difference is determined by the variance of the distribution on $x$. The posterior variance can be reduced to:

Therefore for any fixed generative distribution on $x$, the uncertainty on $w_1$ declines as $𝒪\left\{\frac{1}{N}\right\}$.