Exercise 7.2 - Multi-output linear regression

For multi-output linear regression, if the outputs are independent, then for each output dimension subscripted by $j$, we have:

then:

The independence implies:

Taking its logarithm and saving terms dependent on $\mathbf{\text{𝐖}}$, we have:

Interchanging the order of summation helps to decompose the loss into:

where each:

is but the MLE loss for a 1D linear regression. Thus the columns of $\mathbf{\text{𝐖}}$ can be estimated independently by:

where $\mathbf{\text{𝐗}}$ is the $D\ast N$ design matrix and $\mathbf{\text{𝐘}}$ is a columm matrix with length $N$ that embeds the $j$-th component of the output. This is a little different from the symbols from the textbook but simple $\alpha$-reductions would eliminate such difference. Equation (7.90) is incorrect by missing one design matrix.

As a compact way of writing ${\mathbf{\text{𝐖}}}^{\text{MLE}}$, we would have:

where $\mathbf{\text{𝐘}}$ is a $N\ast M$ matrix.

For the case in this exercise, we have $D=2$, $N=6$, $M=2$:

Thus:

One can observe that two columns for $\mathbf{\text{𝐖}}$ are identical, which is obvious by examing the two columns from $\mathbf{\text{𝐘}}$.