Exercise 2.6

(a)

Apply the error bar in $(2.1)$, i.e. $E_{\mathit{out}}(g)\le E_{\mathit{in}}(g)+\sqrt{\frac{1}{2N}\ln <!--\; FUNCTION\; APPLICATION\; -->\frac{2M}{\delta }}$.
The following calculation shows that error on $E_{\mathit{in}}(g)=0.1151$ and error on $E_{\mathit{test}}(g)=0.096$. So the error bar on in-sample error is higher than the error bar from test error.

(b)

If we reserve more examples for testing, then we’ll have less samples for training. We may end up with a hypothesis that is not as good as we could have arrived if using more training samples. So $E_{\mathit{test}}(g)$ might be way too off even the error bar on it is small.

import numpy as np 
epsilon = 0.05 
 
N = 200 
# test bound 
print('test bound: ', np.sqrt(np.log(2/epsilon)/2/N)) 
 
# train bound 
M = 1000 
N = 400 
print('train bound: ', np.sqrt(np.log(2*M/epsilon)/2/N))

test bound:  0.09603227913199208
train bound:  0.11509037065006825