Homepage › Solution manuals › Kevin P. Murphy › Machine Learning: a Probabilistic Perspective

Kevin P. Murphy

Machine Learning: a Probabilistic Perspective

Machine Learning: a Probabilistic Perspective

A comprehensive introduction to machine learning that uses probabilistic models and inference as a unifying approach.

2 Probability

Exercise 2.1 - Probability are sensitive to the form of the question that was used to generate the answer

Exercise 2.10 - Deriving the inverse gamma density

Exercise 2.11 - Normalization constant for a 1D Gaussian

Exercise 2.12 - Expressing mutual information in terms of entropies

Exercise 2.13 - Mutual information for correlated normals

Exercise 2.14 - A measure of correlation

Exercise 2.15 - MLE minimizes KL divergence to the empirical distribution

Exercise 2.16 - Mean, mode, variance for the beta distribution

Exercise 2.17 - Expected value of the minimum

Exercise 2.2 - Legal reasoning

Exercise 2.3 - Variance of a sum

Exercise 2.4 - Bayes rule for medical diagnosis

Exercise 2.5 - The Monty Hall problem(The dilemma of three doors)

Exercise 2.6 - Conditional Independence

Exercise 2.7 - Pairwise independence does not imply mutual independence

Exercise 2.8 - Conditional independence iff joint factorizes

Exercise 2.9 - Conditional independence

3 Generative models for discrete data

Exercise 3.1 - MLE for the Beroulli/binomial model

Exercise 3.10 - Taxicab problem

Exercise 3.11 - Bayesian analysis of the exponential distribution

Exercise 3.12 - MAP estimation for the Bernoulli with non-conjugate priors

Exercise 3.13 - Posterior predictive distribution for a batch of data with the dirichlet-multinomial model

Exercise 3.14 - Posterior predictive for Dirichlet-multinomial

Exercise 3.15 - Setting the hyper-parameters I

Exercise 3.16 - Setting the beta hyper-parameters II

Exercise 3.17 - Marginal likelihood for beta-binomial under uniform prior

Exercise 3.18 - Bayes factor for coin tossing

Exercise 3.19 - Irrelevant features with naive Bayes

Exercise 3.2 - Marginal likelihood for the Beta-Bernoulli model

Exercise 3.20 - Class conditional densities for binary data

Exercise 3.21 - Mutual information for naive Bayes classifiers with binary features

Exercise 3.22 - Fitting a naive Bayesian spam filter by hand

Exercise 3.3 - Posterior predictive for Beta-Binomial model

Exercise 3.4 - Beta updating from censored likelihood

Exercise 3.5 - Uninformative prior for log-odds ratio

Exercise 3.6 - MLE for the Poisson distribution

Exercise 3.7 - Bayesian analysis of the Poisson distribution

Exercise 3.8 - MLE for the uniform distribution

Exercise 3.9 - Bayesian analysis of the uniform distribution

4 Gaussian models

Exercise 4.1 - Uncorrelated does not imply independent

Exercise 4.10 - Derivation of information form formulae for marginalizing and conditioning

Exercise 4.11 - Derivation of the NIW posterior

Exercise 4.12 - BIC for Gaussians

Exercise 4.13 - Gaussian posterior credible interval

Exercise 4.14 - MAP estimation for 1d Gaussians

Exercise 4.15 - Sequential(recursive) updating of covariance matrix

Exercise 4.16 - Likelihood ratio for Gaussians

Exercise 4.17 - LDA/QDA on height/weight data

Exercise 4.18 - Naive Bayes with mixed features

Exercise 4.19 - Decision boundary for LDA with semi tied covariances

Exercise 4.2 - Uncorrelated and Gaussian does not imply independent unless jointly Gaussian

Exercise 4.20 - Logistic regression vs LDA/QDA

Exercise 4.21 - Gaussian decision boundaries

Exercise 4.22 - QDA with 3 classes

Exercise 4.23 - Scalar QDA

Exercise 4.3 - Correlation coefficient is between -1 and 1

Exercise 4.4 - Correlation coefficient for linearly related variables is 1 or -1

Exercise 4.5 - Normalization constant for a multidimensional Gaussian

Exercise 4.6 - Bivariate Gaussian

Exercise 4.7 - Conditioning a bivariate Gaussian

Exercise 4.8 - Whitening vs standardizing

Exercise 4.9 - Sensor fusion with known variances in 1d

5 Bayesian statistics

Exercise 5.1 - Proof that a mixture of conjugate priors is indeed conjugate

Exercise 5.10 - Decision rule for trading off FPs and FNs

Exercise 5.2 - Optimal threshold on classification probability

Exercise 5.3 - Reject option in classifiers

Exercise 5.4 - More reject options

Exercise 5.5 - Newsvendor problem

Exercise 5.6 - Bayes factors and ROC curves

Exercise 5.7 - Bayes model averaging helps predictive accuracy

Exercise 5.8 - MLE and model selection for a 2d discrete distribution

Exercise 5.9 - Posterior median is optimal estimate under L1 loss

6 Frequentist statistics

Exercise 6.1 - Pessimism of LOOCV

Exercise 6.2 - James Stein estimator for Gaussian means

Exercise 6.3 - $\hat{\sigma}^{2}_{\text{MLE}}$ is biased

Exercise 6.4 - Estimation of $\sigma^{2}$ when $\mu$ is known

7 Linear regression

Exercise 7.1 - Behavior of training set error with increasing sample size

Exercise 7.10 - Bayesian linear regression using the g-prior

Exercise 7.2 - Multi-output linear regression

Exercise 7.3 - Centering and ridge regression

Exercise 7.4 - MLE for $\sigma^{2}$ for linear regression

Exercise 7.5 - MLE for the offset term in linear regression

Exercise 7.6 - MLE for simple linear regression

Exercise 7.7 - Sufficient statistics for online linear regression

Exercise 7.8 - Bayesian linear regression in 1d with known $\sigma^{2}$

Exercise 7.9 - Generative model for linear regression

8 Logistic regression

Exercise 8.1 - Spam classification using logistic regression

Exercise 8.2 - Spam classification using naive Bayes

Exercise 8.3 - Gradient and Hessian of log-likelihood for logistic regression

Exercise 8.4 - Gradient and Hessian of log-likelihood for multinomial logistic regression

Exercise 8.5 - Symmetric version of l2 regularized multinomial logistic regression

Exercise 8.6 - Elementary properties of l2 regularized logistic regression

Exercise 8.7 - Regularizing separate terms in 2d logistic regression

9 Generalized linear models and the exponential family

Exercise 9.1 - Conjugate prior for univariate Gaussian in exponential family form

Exercise 9.2 - The MVN is in the exponential family

10 Directed graphical models(Bayes nets)

Exercise 10.1 - Marginalizing a node in a DGM

Exercise 10.2 - Bayes Ball

Exercise 10.3 - Markov blanket for a DGM

Exercise 10.4 - Hidden variables in DGMs

Exercise 10.5 - Bayes nets for a rainy day

Exercise 10.6 - Fishing nets

Exercise 10.7 - Removing leaves in BN20 networks

Exercise 10.8 - Handling negative findings in the QMR network

Exercise 10.9 - Moralization does not introduce new independence statements

11 Mixture models and the EM algorithm

Exercise 11.1 - Student T as infinite mixture of Gaussian

Exercise 11.10 - Deriving the K-means cost function

Exercise 11.11 - Visible mixtures of Gaussians are in exponential family

Exercise 11.12 - EM for robust linear regression with a Student t likelihood

Exercise 11.13 - EM for EB estimation of Gaussian shrinkage model

Exercise 11.14 - EM for censored linear regression

Exercise 11.15 - Posterior mean and variance of a truncated Gaussian

Exercise 11.2 - EM for mixture of Gaussians

Exercise 11.3 - EM for mixtures of Bernoullis

Exercise 11.4 - EM for mixture of Student distributions

Exercise 11.5 - Gradient descent for fitting GMM

Exercise 11.6 - EM for a finite scale mixture of Gaussians

Exercise 11.7 - Manual calculation of the M step for a GMM

Exercise 11.8 - Moments of a mixture of Gaussians

Exercise 11.9 - K-means clustering by hand

12 Latent linear models

Exercise 12.1 - M-step for FA

Exercise 12.10 - Efficiently evaluating the PPCA density

Exercise 12.11 - PPCA vs FA

Exercise 12.2 - MAP estimation for the FA model

Exercise 12.3 - Heuristic for assessing applicability of PCA

Exercise 12.4 - Deriving the second principal component

Exercise 12.5 - Deriving the residual error for PCA

Exercise 12.6 - Derivation of Fisher's linear discriminant

Exercise 12.7 - PCA via successive deflation

Exercise 12.8 - Latent semantic indexing

Exercise 12.9 - Imputation in a FA model

13 Sparse linear models

Exercise 13.1 - Partial derivative of the RSS

Exercise 13.10 - GSM representation of group lasso

Exercise 13.11 - Projected gradient descent for l1 regularized least squares

Exercise 13.12 - Subderivative of the hinge loss function

Exercise 13.13 - Lower bounds to convex functions

Exercise 13.2 - Derivation of M-step for EB for linear regression

Exercise 13.3 - Derivation of fixed point updates for EB for linear regression

Exercise 13.4 - Marginal likelihood for linear regression

Exercise 13.5 - Reducing elastic net to lasso

Exercise 13.6 - Shrinkage in linear regression

Exercise 13.7 - Prior for the Bernoulli rate parameter in the spike and slab model

Exercise 13.8 - Deriving E step for GSM prior

Exercise 13.9 - EM for sparse probit regression with Laplace prior

14 Kernels

Exercise 14.1 - Fitting an SVM classifier by hand

Exercise 14.2 - Linear separability

15 Gaussian processes

Exercise 15.1 - Reproducing property

16 Adaptive basis function models

Exercise 16.1 - Nonlinear regression for inverse dynamics

17 Markov and hidden Markov models

Exercise 17.1 - Derivation of $Q$ function for HMM

Exercise 17.2 - Two filter approach to smoothing in HMMs

Exercise 17.3 - EM for HMMs with mixture of Gaussian observations

Exercise 17.4 - EM for HMMs with tied mixtures

18 State space models

Exercise 18.1 - Derivation of EM for LG-SSM

Exercise 18.2 - Seasonal LG-SSM model in standard form

19 Undirected graphical models(Markov random fields)

Exercise 19.1 - Derivation of the log partition function

Exercise 19.2 - CI properties of Gaussian graphical models

Exercise 19.3 - Independencies in Gaussian graphical models

Exercise 19.4 - Cost of training MRFs and CRFs

Exercise 19.5 - Full conditional in an Ising model

20 Exact inference for graphical models

Exercise 20.1 - Variable elimination

Exercise 20.2 - Gaussian times Gaussian is Gaussian

Exercise 20.3 - Message passing on a tree

Exercise 20.4 - Inference in 2D lattice MRFs

21 Variational inference

Exercise 21.1 - Laplace approximation to $p(\mu,\log \sigma|\mathcal{D})$ for a univariate Gaussian

Exercise 21.10 - VB for binary FA with probit link

Exercise 21.2 - Laplace approximation to normal-gamma

Exercise 21.3 - Variational lower bound for VB for univariate Gaussian

Exercise 21.4 - Variational lower bound for VB for GMMs

Exercise 21.5 - Derivation of $\mathbb{E}[\log \pi_{k}]$ under a Dirichlet distribution

Exercise 21.6 - Alternative derivation of the mean field updates for the Ising model

Exercise 21.7 - Forwards vs reverse KL divergence

Exercise 21.8 - Derivation of the structured mean field updates for FHMM

Exercise 21.9 - Variational EM for binary FA with sigmoid link

23 Monte Carlo inference

Exercise 23.1 - Sampling from a Cauchy

Exercise 23.2 - Rejection sampling from a Gamma using a Cauchy proposal

Exercise 23.3 - Optimal proposal for particle filtering with linear-Gaussian measurement model

24 Markov chain Monte Carlo (MCMC) inference

Exercise 24.1 - Gibbs sampling from a 2D Gaussian

Exercise 24.2 - Gibbs sampling for a 1D Gaussian mixture model

Exercise 24.3 - Gibbs sampling from the Potts model

Exercise 24.4 - Full conditionals for hierarchical model of Gaussian means

Exercise 24.5 - Gibbs sampling for robust linear regression with a Student t likelihood

Exercise 24.6 - Gibbs sampling for probit regression

Exercise 24.7 - Gibbs sampling for logistic regression with the Student approximation

26 Graphical model structure learning

Exercise 26.1 - Causal reasoning in the sprinkler network

27 Latent variable models for discrete data

Exercise 27.1 - Partition function for an RBM