Exercise 4.3

(a) Consider a given $H$

• If the best approximation from $H$ is less complex than the initial target function, then when we increase the complexity of $f$, the deterministic noise in general should increase, since it’ll be harder for functions in $H$ to fit the target function. There’ll be a higher tendency to overfit.
• If the best approximation from $H$ is more complex than the initial target function, then when we increase the complexity of $f$, the deterministic noise in general may decrease first, reducing the deterministic noise and there’ll be a lower tendency to overfit. But once the complexity of $f$ exceeds the best function approximation from $H$, and if we continue increase the complexity of $f$, we will increase the deterministic noise and thus increase the tendency to overfit.

(b) Given a fixed $f$

• If the best approximation from $H$ is less complex than the target function, then when we decrease the complexity of $H$, we increase the deterministic noise thus increasing the tendency of overfit.
• If the best approximation from $H$ is more complex than the target function, then when we decrease the complexity of $H$, we will decrease the deterministic noise thus decreasing the tendency of overfit. Well, if we continue to decrease the complexity of $H$, passing the point where its complexity is equal to $f$, we start to increase the deministic noise again and thus increasing overfit.