Exercise 4.20 - Logistic regression vs LDA/QDA

The underlying assumptions for all four classifiers are as follows:

• GaussI assumes a covariance matrix as an identity matrix;
• GaussX has no prior assumption on the covariance matrix;
• LinLog assumes that different classes share the same covariance matrix;
• QuadLog has no prior assumption on covariance matrix, yet it assumes that all data from one class are subject to a normal distribution;

From the perspective of complexity we have the following order:

QuadLog $=$GaussX $>$LinLog $>$GaussI.

The MLE likelihood should follow the same order, this answers the question (a)-(d).

For question (e), the argument is untrue in general. For example, model $M$ predicts two samples belonging to the first class with probability vectors $(0.49,0.51)$ and $(0.99,0.1)$. While $M^{\prime }$ outputs $(0.51,0.49)$ and $(0.51,0.49)$. Now $M^{\prime }$ is correct on both samples so $R(M)>R(M^{\prime })$, but:

so $L(M)>L(M^{\prime })$, this is sufficient for disproving the argument.