Exercise 13.6 - Shrinkage in linear regression

For the ordinary least square, the loss is defined as:

Since $\mathbf{\text{𝐗}}{\mathbf{\text{𝐗}}}^{\text{T}}=\mathbf{\text{𝐈}}$:

Take its derivative w.r.t. $w_k$, where we note that all $D$ weights has been decoupled in the $\text{RSS}$:

Therefore we ends up with:

For the ridge regression:

Take its derivative and set it to zero:

Thus

Finally, recall that

Observe Figure 13.24, it is easy to address the black line as OLS, the gray one Ridge and the dotted one lasso. Obviously ${\lambda }_1={\lambda }_2=1$. It is noticeable that ridge cause a shrinkage to horizontal axis while lasso cause a sharp shrinkage to zero under certain threshold.