Exercise 3.2 - Marginal likelihood for the Beta-Bernoulli model

This exercise continues the discussion of the toy coin-toss experiment, so we borrow all symbols from the exercise above. The likelihood takes the form:

The prior distribution of $𝜃$ takes the form:

where we adopt $C_1(a,b)$ in the hope of eliminating the ambiguity of using $\propto$, which, although simplifies the symbolization, results in countless errors.

The posterior distribution takes the form:

The first step is the straightforward Bayesian rule, the second is the Markov property. In the last step, we adopt the equations before. Since $p(𝜃|𝒟,a,b)$ should be normalized w.r.t. $𝜃$, it has to be a Beta distribution with hyperparameters $N_1+a,N_0+b$. We can now derive the evidence of $𝒟$ w.r.t. $a$ and $b$ explicitly. The normalization of $p(𝜃|𝒟,a,b)$ indicates that:

so:

where $C_1(\cdot ,\cdot )$ is the normalization factor for the Beta distribution. This is enough for deriving (3.80) by recalling the normalization of Beta distribution. The value of $p(𝒟|a,b)$ can help us select proper hyperparametes.

As for prediction:

The first step is the Bayesian rule, the second is Markov property. The rest is straightforward algebra.

Concretely, we calcualte $p(𝒟)$ where $𝒟=\left\{1,0,0,1,1\right\}$:

Rename the variables $\alpha =a+b,{\alpha }_1=a,{\alpha }_0=b$, we have (3.83). To derive (3.80), we make use of: