Solution to Exercise 3.4 from „Learning from Data“

We have

y = X w^{∗} + 𝜖

, take this into the expression for in-sample estimate of

y

is

ŷ

, we have

\begin{array}{l} ŷ & = Hy \\ = H (X w^{∗} + 𝜖) \\ = Hx w^{∗} + H𝜖 \\ = X {(X^{T} X)}^{- 1} X^{T} X w^{∗} + H𝜖 \\ = X w^{∗} + H𝜖 \end{array}

The in-sample error vector is:

\begin{array}{l} ŷ - y & = X w^{∗} + H𝜖 - (X w^{∗} + 𝜖) \\ = (H - I) 𝜖 \end{array}

\begin{array}{l} E_{in} (w_{lin}) & = \frac{1}{N} ∥ X w_{lin} - y ∥^{2} \\ = \frac{1}{N} ∥ y - ŷ ∥^{2} \\ = \frac{1}{N} ∥ (I - H) 𝜖 ∥^{2} \\ = \frac{1}{N} 𝜖^{T} {(I - H)}^{T} (I - H) 𝜖 \\ = \frac{1}{N} 𝜖^{T} (I - H^{T}) (I - H) 𝜖 \\ = \frac{1}{N} 𝜖^{T} (I - H) (I - H) 𝜖 \\ = \frac{1}{N} 𝜖^{T} (I - H) 𝜖 \end{array}

\begin{array}{l} E_{D} [E_{in} (w_{lin})] & = E_{D} [\frac{1}{N} 𝜖^{T} (I - H) 𝜖] \\ = \frac{1}{N} (E_{D} [𝜖^{T} 𝜖] - E_{D} [𝜖^{T} H𝜖]) \\ = \frac{1}{N} (E_{D} [\sum_{k = 1}^{N} 𝜖_{k}^{2}] - E_{D} [\sum_{i = 1}^{N} \sum_{j = 1}^{N} 𝜖_{i} h_{ij} 𝜖_{j}]) \\ = \frac{1}{N} (\sum_{k = 1}^{N} E_{D} 𝜖_{k}^{2} - \sum_{i = 1}^{N} \sum_{j = 1}^{N} E_{D} [𝜖_{i} h_{ij} 𝜖_{j}]) \\ = \frac{1}{N} (N σ^{2} - \sum_{i = 1}^{N} E_{D} [𝜖_{i}^{2} h_{ii}]) \\ = \frac{1}{N} (N σ^{2} - \sum_{i = 1}^{N} h_{ii} E_{D} [𝜖_{i}^{2}]) \\ = \frac{1}{N} (N σ^{2} - σ^{2} trace (H)) \\ = σ^{2} (1 - \frac{trace (H)}{N}) \\ = σ^{2} (1 - \frac{d + 1}{N}) \end{array}

Here we used the independence of $𝜖_{i}$ , and also we assume $H$ is not random variable, so we are not doing expectation w.r.t. $x$ , that’s how we can pull $h_{ii}$ out of $E_{D} [𝜖_{i}^{2} h_{ii}]$ .

Since

X

doesn’t change, only

𝜖

changes, we have

\begin{array}{l} ŷ - y^{'} & = X w^{∗} + H𝜖 - (X w^{∗} + 𝜖^{'}) \\ = H𝜖 - 𝜖^{'} \end{array}

Following the procedure in problems (c) and (d), we have

\begin{array}{l} E_{D, 𝜖^{'}} [E_{test} (w_{lin})] & = E_{D, 𝜖^{'}} [\frac{1}{N} ∥ y^{'} - ŷ ∥^{2}] \\ = E_{D, 𝜖^{'}} [\frac{1}{N} ∥ 𝜖^{'} - H𝜖 ∥^{2}] \\ = \frac{1}{N} E_{D, 𝜖^{'}} [{(𝜖^{'} - H𝜖)}^{T} (𝜖^{'} - H𝜖)] \\ = \frac{1}{N} E_{D, 𝜖^{'}} [(𝜖^{′T} - 𝜖^{T} H^{T}) (𝜖^{'} - H𝜖)] \\ = \frac{1}{N} E_{D, 𝜖^{'}} [(𝜖^{′T} - 𝜖^{T} H) (𝜖^{'} - H𝜖)] \\ = \frac{1}{N} E_{D, 𝜖^{'}} [𝜖^{′T} 𝜖^{'} - 𝜖^{′T} H𝜖 - 𝜖^{T} H 𝜖^{'} + 𝜖^{T} HH𝜖] \\ = \frac{1}{N} E_{D, 𝜖^{'}} [𝜖^{′T} 𝜖^{'} - 𝜖^{′T} H𝜖 - 𝜖^{T} H 𝜖^{'} + 𝜖^{T} H𝜖] \\ = \frac{1}{N} E_{D, 𝜖^{'}} [𝜖^{′T} 𝜖^{'} + 𝜖^{T} H𝜖] \\ = \frac{1}{N} (\sum_{k = 1}^{N} E_{D} 𝜖_{k}^{' 2} + \sum_{i = 1}^{N} \sum_{j = 1}^{N} E_{D} [𝜖_{i} h_{ij} 𝜖_{j}]) \\ = σ^{2} (1 + \frac{d + 1}{N}) \end{array}

Where we have used the fact that $𝜖$ and $𝜖^{'}$ are independent of each other and each $𝜖_{k}$ and $𝜖_{k}^{'}$ are independent among themselves.

Exercise 3.4

Answers

Comments

Exercise 3.4

Answers

Comments

Add answer