Exercise 27.1 - Partition function for an RBM

Answers

The partition function for a binary RBM takes the form:

Z (𝜃) = \sum_{𝐯} \sum_{𝐡} \exp {𝐯^{T} 𝐖 𝐡 + 𝐯^{T} 𝐛 + 𝐡^{T} 𝐜} .

For a fixed $𝐯$ , the terms to be summed up are:

\exp {𝐯^{T} 𝐛} \cdot \sum_{𝐡} \exp {𝐡^{T} 𝐮},

where:

𝐮 = 𝐖^{T} 𝐯 + 𝐜 .

The summation $\sum_{𝐡} \exp {𝐡^{T} 𝐮}$ shall be written as:

\begin{aligned} \exp (0) + \exp (u_{1}) + \exp (u_{2}) + \exp (u_{2} + u_{1}) + \dots \\ = & \exp (0) + \exp (u_{1}) + \exp (u_{2}) [\exp (0) + \exp (u_{1})] + \dots, \end{aligned}

Hence:

\sum_{𝐡} \exp {𝐡^{T} 𝐮} = \prod_{k = 1}^{K} (1 + \exp (u_{k})),

whose computation cost is $𝒪 (K)$ . The overall complexity is thus $𝒪 (K \cdot 2^{R})$ . Since the summation order for $𝐯$ and $𝐡$ is interchangable, the complexity is reduced to:

𝒪 (\min_{R, K} {K \cdot 2^{R}, R \cdot 2^{K}}) .

solour_lfq

2021-03-24 13:42

Exercise 27.1 - Partition function for an RBM

Answers

Comments

Add answer