Exercise 13.1 - Partial derivative of the RSS

Answers

For question (a), define:

RSS (𝐰) = \sum_{n = 1}^{N} {(y_{n} - 𝐰^{T} 𝐱_{n})}^{2} .

Then we have straightforwardly:

\begin{align} \frac{\partial}{\partial w_{j}} RSS (𝐰) = & \sum_{n = 1}^{N} 2 \cdot (y_{n} - 𝐰^{T} 𝐱_{n}) (- x_{𝑛𝑗}) \\ = & - \sum_{n = 1}^{N} 2 \cdot (x_{𝑛𝑗} y_{n} - x_{𝑛𝑗} \sum_{i = 1}^{D} w_{i} x_{𝑛𝑖}) \\ = & - \sum_{n = 1}^{N} 2 (x_{𝑛𝑗} y_{n} - x_{𝑛𝑗} \sum_{i \neq j}^{D} w_{i} x_{𝑛𝑖} - x_{𝑛𝑗}^{2} w_{j}) . \end{align}

From which we observe that $w_{j}$ ’s coefficient is:

a_{j} = 2 \sum_{n = 1}^{N} x_{𝑛𝑗}^{2},

while the rest irrelevent terms can be absorbed into:

c_{j} = 2 \sum_{n = 1}^{N} x_{𝑛𝑗} (y_{n} - 𝐰_{- j}^{T} 𝐱_{n, - j}) .

The optimal value for $w_{j}$ is:

ŵ_{j} = \frac{c_{j}}{a_{j}} .

For question (b), (13.184) is obvious by plugging the definition of $𝐫_{k}$ into (13.182)-(13.183) and the expression for $ŵ_{j}$ .

solour_lfq

2021-03-24 13:42