Exercise 3.12 - MAP estimation for the Bernoulli with non-conjugate priors

For question (a), we adopt the different prior:

The posterior distribution now reads:

whose support is $\left\{0.4,0.5\right\}$, so the MAP is:

For question (b), it is intuitive that the non-conjugate has better performance when $N$ is small. But the conjugate Bayesian method prevails with $N$ grows. For a solid verification, consider the case where $N$ is large, the probability that $\frac{N_1}{N}$ deviates $𝜖$ from $0.41$ can be bounded by the Chernoff bounding. Let ${\left\{{\xi }_n\right\}}_{n=1}^N$ be a collection of i.i.d. Bernoulli random variables with distribution:

Denote $X={\sum <!--\; FUNCTION\; APPLICATION\; -->}_{n=1}^N{\xi }_n$ as the random variable marks their summation. Then:

Where $\lambda$ can be an arbitrary positive number. The probability that $X$ exceeds $N\cdot (0.41+𝜖)$, denoted by $P_1$ is bounded by the lower bound of the last line in the deduction above. With $N=1000$, $𝜖=0.05$, the probability is numerically bounded by $0.00602$. The other side of error $\text{Pr}(X\le N\cdot (0.41-𝜖))$, whose probability is $P_2$, can be derived in a similar way. The probability that the Bayesian way is dominated by the non-uniform prior is no higher than $P_1+P_2$. Taking $𝜖\to 0.1$ and $N\to \infty$, this bound remains negligible.