Exercise 12.3 - Heuristic for assessing applicability of PCA

We derive this heuristics from an information theory’s perspective. Recall that the differential entropy for a MVN is (with ${\sigma }_i^2={\lambda }_i^{-1}$):

After PCA, the covariance for this MVN model is obatined by replacing the smallest $d^{\prime }$ variances into ${\sigma }^2\to 0$, hence the difference in entropy is:

For two eigen series with the same mean $\bar{\lambda }$, it is plausible to expect that the product of the largest $d^{\prime }$ values in the series with a larger variance is larger, hence the information loss is smaller, making the PCA better regarding information compression.