Exercise 21.1 - Laplace approximation to $p(\mu,\log \sigma|\mathcal{D})$ for a univariate Gaussian

Answers

Laplace approximation aims at representing $f (μ, l) = \log p (μ, l = \log σ | D)$ from its first and second-order gradients. We have:

\begin{aligned} \log p (μ, l | 𝒟) = & \log p (μ, l, 𝒟) - \log p (𝒟) \\ = & \log p (μ, l) + \log p (𝒟 | μ, l) + const \\ = & \log p (𝒟 | μ, l) + const \\ = & \sum_{n = 1}^{N} \log \frac{1}{\sqrt{2 π σ^{2}}} \cdot \exp {- \frac{{(y_{n} - μ)}^{2}}{2 σ^{2}}} + const \\ = & - N \log σ + \sum_{n = 1}^{N} - \frac{{(y_{n} - μ)}^{2}}{2 σ^{2}} + const \\ = & - N \cdot l - \frac{\exp (- 2 l)}{2} \sum_{n = 1}^{N} {(y_{n} - μ)}^{2} + const . \end{aligned}

Thus we can take gradients:

\begin{align} \frac{\partial \log p (μ, l | 𝒟)}{∂𝜇} = & \frac{- \exp (- 2 l)}{2} \sum_{n = 1}^{N} 2 \cdot (y_{n} - μ) \\ = & - \frac{N}{σ^{2}} \cdot (ȳ - μ), \\ \frac{\partial \log p (μ, l | 𝒟)}{∂𝑙} = & - N - \frac{- 2 \cdot \exp (- 2 l)}{2} \sum_{n = 1}^{N} {(y_{n} - μ)}^{2} \\ = & - N + \frac{1}{σ^{2}} \sum_{n = 1}^{N} {(y_{n} - μ)}^{2}, \\ \frac{\partial^{2} \log p (μ, l | 𝒟)}{\partial μ^{2}} = & - \frac{N}{σ^{2}}, \\ \frac{\partial^{2} \log p (μ, l | 𝒟)}{\partial l^{2}} = & - \frac{2}{σ^{2}} \sum_{n = 1}^{N} {(y_{n} - μ)}^{2}, \\ \frac{\partial^{2} \log p (μ, l | 𝒟)}{∂𝜇∂𝑙} = & N \cdot (ȳ - μ) \cdot (- 2) \cdot \frac{1}{σ^{2}} . \end{align}

Finally, to conduct the Laplace approximation, recall that:

f (μ, l) \approx const + (\begin{matrix} \frac{∂𝑓}{∂𝜇} & \frac{∂𝑓}{∂𝑙} \end{matrix}) (\begin{matrix} μ \\ l \end{matrix}) + \frac{1}{2} (\begin{matrix} μ & l \end{matrix}) 𝐇 (\begin{matrix} μ \\ l \end{matrix}) .

With this expansion in mind, we have:

p (μ, l) = \exp (l (μ, l)) = const \cdot \exp {- \frac{1}{2} (\begin{matrix} μ - m_{1} & l - m_{2} \end{matrix}) Λ (\begin{matrix} μ - m_{1} \\ l - m_{2} \end{matrix})} .

Calibrating the coefficient, we know that the covariance and the mean for this approximation are:

Σ = - {(\begin{matrix} \frac{\partial^{2} \log p (μ, l | D)}{\partial μ^{2}} & \frac{\partial^{2} \log p (μ, l | D)}{\partial l^{2}} \\ \frac{\partial^{2} \log p (μ, l | D)}{\partial l^{2}} & \frac{\partial^{2} \log p (μ, l | D)}{∂𝜇∂𝑙} \end{matrix})}^{- 1},

m = (\begin{matrix} \frac{\partial^{2} \log p (μ, l | D)}{\partial μ^{2}} & \frac{\partial^{2} \log p (μ, l | D)}{\partial l^{2}} \\ \frac{\partial^{2} \log p (μ, l | D)}{\partial l^{2}} & \frac{\partial^{2} \log p (μ, l | D)}{∂𝜇∂𝑙} \end{matrix}) (\begin{matrix} \frac{∂𝑓}{∂𝜇} \\ \frac{∂𝑓}{∂𝑙} \end{matrix}) .

solour_lfq

2021-03-24 13:42