Exercise 21.7 - Forwards vs reverse KL divergence

To formulate $\mathit{𝕂𝕃}(p\parallel q)$ with $q(x,y)=q(x)q(y)$, we have:

The non-negativity of KL divergence implies that we should set $q(x)=p(x)$ and $q(y)=p(y)$.

For the tabular case, let $q={\left\{q_{\mathit{𝑖𝑗}}\right\}}_{i,j=1}^4$ be the variational distribution s.t.:

with six free parameters: $q_x(1)$, $q_x(2)$, $q_x(3)$, $q_y(1)$, $q_y(2)$ and $q_y(3)$. The reverse KL is:

Since the decomposition assumes the independence between $x$ and $y$, we have:

Meanwhile, the symmetry within the table indicates that the roles of $q_x(1)$ and $q_x(2)$ are interchangable, so are $q_x(3)$ and $q_x(4)$, $q_y(1)$ and $q_y(2)$, $q_y(3)$ and $q_y(4)$. This observation, together with the property of the entropy (and the Cauchy inequality), implies that $q_x$ takes the form:

So is $q_y$, hence the KL takes the value:

where $\alpha$ takes some value between $\frac{1}{8}$ and $\frac{1}{4}$. In this case the minimum is not necessarily unique since the function to be minimized is the sum of a convex and a concave function w.r.t. $\alpha$.