Exercise 8.3 - Gradient and Hessian of log-likelihood for logistic regression

Answers

For question (a),

\frac{d}{d a} σ (a) = \frac{\exp (- a)}{{(1 + \exp (- a))}^{2}} = \frac{1}{1 + e^{- a}} \frac{e^{- a}}{1 + e^{- a}} = σ (a) \cdot (1 - σ (a)) .

For question (b),

\begin{align} g (𝐰) = & \frac{\partial}{\partial 𝐰} NLL (𝐰) \\ = & - \sum_{n = 1}^{N} \frac{\partial}{\partial 𝐰} [y_{i} \log μ_{i} + (1 - y_{i}) \log (1 - μ_{i})] \\ = & - \sum_{n = 1}^{N} y_{i} \frac{1}{σ_{i}} σ_{i} (1 - σ_{i}) \cdot 𝐱_{i} + (1 - y_{i}) \frac{- 1}{1 - σ_{i}} σ (1 - σ_{i}) \cdot 𝐱_{i} \\ = & \sum_{n = 1}^{N} (σ (𝐰^{T} 𝐱_{i}) - y_{i}) 𝐱_{i}, \end{align}

where $σ_{i} = σ (𝐰^{T} 𝐱_{i})$ .

For question (c), the result is obvious. For an arbitrary vector $𝐮$ :

\begin{aligned} 𝐮^{T} 𝐇 𝐮 & = {(𝐗 𝐮)}^{T} 𝐒 (𝐗 𝐮) \\ = 𝐯^{T} 𝐒 𝐯 \\ = \sum_{d = 1}^{D} v_{d}^{2} \cdot μ_{d} \cdot (1 - μ_{i}) \geq 0 . \end{aligned}

Hence $𝐇$ is positive definite if all $μ_{i}$ are within $(0, 1)$ , otherwise it is semi-positive definite.

solour_lfq

2021-03-24 13:42