Exercise 8.13

Consider a data set with two positive examples at $x_1=(0,0)$ and $x_2=(1,0)$, and one negative example at $x_3=(0,1)$. We look for hyperplane (line) that separate the negative example with the positive examples. As there’s only 1 negative example, it has to be the support vector, either one of the two positive examples or both of them can be the support vectors. It’s not hard by trial and error to find out that the optimal fat-hyperplane is $-2{[x]}_2+1=0$, i.e. with $(b,w)=(1,[0,-2])$

The optimal solution ${\alpha }^{\text{∗}}$ has to satisfy $w^{\text{∗}}={\sum <!--\; FUNCTION\; APPLICATION\; -->}_{n=1}^N{y}_n{\alpha }_n^{\text{∗}}{x}_n$,

since

we have ${\alpha }_1=0$.

On the other hand, for this hyperplane, all three points are support vectors. It’s easy to check that for $n=1,2,3$, we have

So if a point $(x_n,y_n)$ is on the boundary satisfying $y_n({(w^{\text{∗}})}^T{x}_n+b^{\text{∗}})=1$, it’s possible that ${\alpha }_n^{\text{∗}}=0$ as the ${\alpha }_1^{\text{∗}}=0$ here.