Exercise 12.4 - Deriving the second principal component

For this exercise, minimizing $J$ makes ${\mathbf{\text{𝐯}}}_1$ and ${\mathbf{\text{𝐯}}}_2$ the first and second principal component for the dataset. By definition, $z_{i,j}$ (where $j=1,2$) is the projection of ${\mathbf{\text{𝐱}}}_i$ onto ${\mathbf{\text{𝐯}}}_j$.

For question (a), $J({\mathbf{\text{𝐯}}}_1,{\mathbf{\text{𝐯}}}_2)$ is the reconstruction loss measured by $l_2$ norm, hence we would have:

from the physics of projection. On the other hand, we could use a more mathematical way for deduction, with:

we have:

Using the fact that ${\mathbf{\text{𝐯}}}_2^{\text{T}}{\mathbf{\text{𝐯}}}_2=1$ and ${\mathbf{\text{𝐯}}}_2^{\text{T}}{\mathbf{\text{𝐯}}}_1=0$, we arrive at:

For question (b), with:

we adopt straightforward matrix algebra:

where we have assumed that $\mathbf{\text{𝐂}}$ is symmetric and semi-positive definite. The next step is to decompose ${\mathbf{\text{𝐯}}}_2$ along the eigenvectors of $\mathbf{\text{𝐂}}$ as:

where $d$ is arranged in the inverse order of the eigenvalues $y_d$, i.e., ${\mathbf{\text{𝐮}}}_1={\mathbf{\text{𝐯}}}_1$, $f_1={\lambda }_1.$ Taking partial gradient w.r.t. ${\lambda }_2$ and ${\lambda }_{12}$ implies:

Hence, taking the gradient w.r.t. ${\mathbf{\text{𝐯}}}_2$ as zero is tantamount to write:

Recall that the eigenvectors for $\mathbf{\text{𝐂}}$ are orthogonal to each other, hence we have:

These equations tell that ${\lambda }_{12}=0$, and ${\lambda }_2$ equals one eigenvalue of $\mathbf{\text{𝐂}}$, $y_d$, whose corresponding eigenvector ${\mathbf{\text{𝐮}}}_d$ is the optimal value for ${\mathbf{\text{𝐯}}}_2$. (We ignore the degenerate case here, but the generalization is straightforward.) For such ${\mathbf{\text{𝐯}}}_2$, the value of $\tilde{J}$ is $-y_d^2$, whose minimum is reached when ${\mathbf{\text{𝐯}}}_2$ is the eigenvector corresponding to the second largest eigenvalue.