Exercise 2.13 - Mutual information for correlated normals

Answers

We have:

I ( X 1 ; X 2 ) = H ( X 1 ) − H ( X 1 | X 2 ) = H ( X 1 ) + H ( X 2 ) − H ( X 1 , X 2 ) = 1 2 log ⁡ 2 π σ 2 + 1 2 log ⁡ 2 π σ 2 + 1 2 log ⁡ ( 2 π ) 2 σ 4 ( 1 − ρ 2 ) = − 1 2 log ⁡ ( 1 − ρ 2 ) .

Here we incorporate a comprehensive deduction on (2.138) and (2.139), which shall not be taken for granted. The differential entropy for a 1D-Gaussian with density function:

p ( x ) = 1 2 π σ 2 exp ⁡ { − x 2 2 σ 2 }

is

− ∫ 1 2 π σ 2 exp ⁡ { − x 2 2 σ 2 } ln ⁡ ( 1 2 π σ 2 exp ⁡ { − x 2 2 σ 2 } ) d x = ln ⁡ ( 2 π σ 2 ) + 1 2 σ 2 𝔼 [ x 2 ] = 1 2 ln ⁡ ( 2 𝜋𝑒 σ 2 ) .

For the multi-dimensional case, we begin by diagonalizing the covariance matrix/decoupling the components and integrating along each independent component. Under this new set of coordinates v 1 , ⋯ , v d , the logarithm of the density can be decomposed into

C + ∑ i = 1 d − v i 2 2 σ i 2 ,

where σ i 2 is the i -th diagonal component in the transformed covariance matrix. The product of all diagonal components is exactly det Σ , hence proving (2.138).

User profile picture
2021-03-24 13:42
Comments