Exercise 12.7 - PCA via successive deflation

The matrix:

is unknown as the projection matrix that maps an arbitrary vector into the subspace that is orthogonal to $\mathbf{\text{𝐯}}$, since:

The only requirement is that $\mathbf{\text{𝐯}}$ is a unit vector: ${\mathbf{\text{𝐯}}}^{\text{T}}\mathbf{\text{𝐯}}=1$.

For generalization to the projection matrix to the supplementary of a multi-dimensional space spanned by a set of orthogonal and unit bases $\left\{{\mathbf{\text{𝐯}}}_1,\cdots ,{\mathbf{\text{𝐯}}}_M\right\}$ (otherwise using the Schmidt procedure first), we only have to use:

For question (a), we have:

For question (b), let $\widetilde{\mathbf{\text{𝐂}}}=\frac{1}{N}\widetilde{\mathbf{\text{𝐗}}}{\widetilde{\mathbf{\text{𝐗}}}}^{\text{T}}$ then its principal eigenvector $\mathbf{\text{𝐮}}$ satisfies:

To solve this equation, expand $\mathbf{\text{𝐮}}$ onto the eigenvectors of $\mathbf{\text{𝐂}}$:

then:

Then the eigenequation becomes:

s.t.:

To solve for this system, we have:

Hence the solution is $d^{\prime }=2$, this finishes the proof.

For question (c), the procedure is roughly:

1.

$[\lambda ,\mathbf{\text{𝐮}}]=f(\mathbf{\text{𝐂}})$;

2.

Save $\lambda$ and $\mathbf{\text{𝐮}}$;

3.

$\mathbf{\text{𝐂}}-=\lambda \mathbf{\text{𝐮}}{\mathbf{\text{𝐮}}}^T$;

4.

Repeat.