Homepage › Solution manuals › Kevin P. Murphy › Machine Learning: a Probabilistic Perspective › Exercise 2.12 - Expressing mutual information in terms of entropies

Machine Learning: a Probabilistic Perspective

Exercise 2.12 - Expressing mutual information in terms of entropies

Answers

We have:

\begin{align} I (X; Y) = & \sum_{x, y} p (x, y) \log \frac{p (x, y)}{p (x) p (y)} \\ = & \sum_{x, y} p (x, y) \log \frac{p (x | y)}{p (x)} \\ = & \sum_{x, y} p (x, y) \log p (x | y) - \sum_{x} (\sum_{y} p (x, y)) \log p (x) \\ = & - H (X | Y) + H (X) . \end{align}

Inversing $X$ and $Y$ yields another formula. One can proceed to show that $I (X; Y) = H (X) + H (Y) - H (X, Y)$ .

solour_lfq

2021-03-24 13:42

Comments