Exercise 7.11

Take derivative w.r.t. $w_{\mathit{ij}}^{(l)}$ in the second term of $E_{\mathit{aug}}(w,\lambda )$, we have its derivative equals to

This proves the equation.

We use the ratio of gradient versus weight to check the rate of decay.

From the derivative, we check the ratio of the second term to the weight $w_{\mathit{ij}}^{(l)}$, and we have $\frac{2\lambda }{N}\frac{1}{{\left(1+{(w_{\mathit{ij}}^{(l)})}^2\right)}^2}$, which achieves maximum value of 1 when $w_{\mathit{ij}}^{(l)}\to 0$. So the smaller the weight, the larger the decay w.r.t. itself.

This indicates that small weights decay much faster than large ones.