CMU 11-785 L21 Boltzmann machines2

The Hopfield net as a distribution

The Helmholtz Free Energy of a System

• At any time, the probability of finding the system in state $s$ at temperature $T$ is $P_T(s)$

• At each state it has a potential energy $E_s$

• The internal energy of the system, representing its capacity to do work, is the average

  • $$U_T=\sum _S{P}_T(s)E_S$$

• The capacity to do work is counteracted by the internal disorder of the system, i.e. its entropy

  • $$H_T=-\sum _S{P}_T(s)\log P_T(s)$$

• The Helmholtz free energy of the system measures the useful work derivable from it and combines the two terms

  • $$F_T=U_T+kT{H}_T$$
  • $$=\sum _S{P}_T(s)E_S-kT\sum _S{P}_T(s)\log P_T(s)$$

• The probability distribution of the states at steady state is known as the Boltzmann distribution

  • Minimizing this w.r.t $P_T(s)$, we get

  • $$P_T(s)=\frac{1}{Z}\exp \left(\frac{-E_S}{kT}\right)$$

  • $Z$ is a normalizing constant

Hopfield net as a distribution

• $E(S)=-{\sum }_{i<j}{w}_{ij}{s}_i{s}_j-b_i{s}_i$
• $P(S)=\frac{\exp (-E(S))}{{\sum }_{S^{\prime }}\exp \left(-E\left(S^{\prime }\right)\right)}$
• The stochastic Hopfield network models a probability distribution over states
• It is a generative model: generates states according to $P(S)$

The field at a single node

• Let’s take one node as example

• Let $S$ and $S^{\prime }$ be the states with the +1 and -1 states

  • $P(S)=P\left(s_i=1\mid s_{j\ne i}\right)P\left(s_{j\ne i}\right)$
  • $P\left(S^{\prime }\right)=P\left(s_i=-1\mid s_{j\ne i}\right)P\left(s_{j\ne i}\right)$
  • $\log P(S)-\log P\left(S^{\prime }\right)=\log P\left(s_i=1\mid s_{j\ne i}\right)-\log P\left(s_i=-1\mid s_{j\ne i}\right)$
  • $\log P(S)-\log P\left(S^{\prime }\right)=\log \frac{P\left(s_i=1\mid s_{j\ne i}\right)}{1-P\left(s_i=1\mid s_{j\ne i}\right)}$

• $\log P(S)=-E(S)+C$

  • $E(S)=-\frac{1}{2}\left(E_{\text{not }i}+{\sum }_{j\ne i}{w}_{ij}{s}_j+b_i\right)$
  • $E\left(S^{\prime }\right)=-\frac{1}{2}\left(E_{\text{not }i}-{\sum }_{j\ne i}{w}_{ij}{s}_j-b_i\right)$

• $\log P(S)-\log P\left(S^{\prime }\right)=E\left(S^{\prime }\right)-E(S)={\sum }_{j\ne i}{w}_{ij}{S}_j+b_i$

  • $\log \left(\frac{P\left(s_i=1\mid s_{j\ne i}\right)}{1-P\left(s_i=1\mid s_{j\ne i}\right)}\right)={\sum }_{j\ne i}{w}_{ij}{s}_j+b_i$

  • $P\left(s_i=1\mid s_{j\ne i}\right)=\frac{1}{1+e^{-\left({\sum }_{j\ne i}{w}_{ij}{s}_j+b_i\right)}}$

• The probability of any node taking value 1 given other node values is a logistic

Redefining the network

• Redefine a regular Hopfield net as a stochastic system
• Each neuron is now a stochastic unit with a binary state $s_i$, which can take value 0 or 1 with a probability that depends on the local field
  • $z_i={\sum }_j{w}_{ij}{s}_j+b_i$
  • $P\left(s_i=1\mid s_{j\ne i}\right)=\frac{1}{1+e^{-z_i}}$
• Note
  • The Hopfield net is a probability distribution over binary sequences (Boltzmann distribution)
  • The conditional distribution of individual bits in the sequence is a logistic
• The evolution of the Hopfield net can be made stochastic
  • Instead of deterministically responding to the sign of the local field, each neuron responds probabilistically
• Recall patterns

The Boltzmann Machine

• The entire model can be viewed as a generative model
• Has a probability of producing any binary vector $y$
  • $E(\mathbf{y})=-\frac{1}{2}{\mathbf{y}}^T\mathbf{W}\mathbf{y}$
  • $P(\mathbf{y})=\mathrm{Cexp}\left(-\frac{E(\mathbf{y})}{T}\right)$
• Training a Hopfield net: Must learn weights to “remember” target states and “dislike” other states
  • Must learn weights to assign a desired probability distribution to states
  • Just maximize likelihood

Maximum Likelihood Training

• $\log (P(S))=\left({\sum }_{i<j}{w}_{ij}{s}_i{s}_j\right)-\log \left({\sum }_{S^{\prime }}\exp \left({\sum }_{i<j}{w}_{ij}{s}_i^{\prime }{s}_j^{\prime }\right)\right)$

• $\mathcal{L}=\frac{1}{N}{\sum }_{S\in \mathbf{S}}\log (P(S))=\frac{1}{N}{\sum }_S\left({\sum }_{i<j}{w}_{ij}{s}_i{s}_j\right)-\log \left({\sum }_{S^{\prime }}\exp \left({\sum }_{i<j}{w}_{ij}{s}_i^{\prime }{s}_j^{\prime }\right)\right)$

• Second term derivation

  • $\frac{d\log \left({\sum }_{S^{\prime }}\exp \left({\sum }_{i<j}{w}_{ij}{s}_i^{\prime }{s}_j^{\prime }\right)\right)}{d{w}_{ij}}={\sum }_{S^{\prime }}\frac{\exp \left({\sum }_{i<j}{w}_{ij}{s}_i^{\prime }{s}_j^{\prime }\right)}{{\sum }_{S^{\prime }}\exp \left({\sum }_{i<j}{w}_{ij}{s}_i^{\prime \prime }{s}_j^{\prime }\right)}{s}_i^{\prime }{s}_j^{\prime }$
  • $\frac{d\log \left({\sum }_{S^{\prime }}\exp \left({\sum }_{i<j}{w}_{ij}{s}_i^{\prime }{s}_j^{\prime }\right)\right)}{d{w}_{ij}}={\sum }_{S_{\prime }}P\left(S^{\prime }\right)s_i^{\prime }{s}_j^{\prime }$
  • The second term is simply the expected value of $s_i{S}_j$, over all possible values of the state
  • We cannot compute it exhaustively, but we can compute it by sampling!

• Overall gradient ascent rule

  • $w_{ij}=w_{ij}+\eta \frac{d⟨\log (P(\mathbf{S}))⟩}{d{w}_{ij}}$

• Overall Training

  • Initialize weights
  • Let the network run to obtain simulated state samples
  • Compute gradient and update weights
  • Iterate

• Note the similarity to the update rule for the Hopfield network

  • The only difference is how we got the samples

Adding Capacity

• Visible neurons

  • The neurons that store the actual patterns of interest

• Hidden neurons

  • The neurons that only serve to increase the capacity but whose actual values are not important

• We could have multiple hidden patterns coupled with any visible pattern

  • These would be multiple stored patterns that all give the same visible output

• We are interested in the marginal probabilities over visible bits

  • $S=(V,H)$
  • $P(S)=\frac{\exp (-E(S))}{{\sum }_{S^{\prime }}\exp \left(-E\left(S^{\prime }\right)\right)}$
  • $P(S)=P(V,H)$
  • $P(V)={\sum }_{H}P(S)$

• Train to maximize probability of desired patterns of visible bits

  • $E(S)=-{\sum }_{i<j}{w}_{ij}{s}_i{s}_j$
  • $P(S)=\frac{\exp \left({\sum }_{i<j}{w}_{ij}{s}_i{s}_j\right)}{{\sum }_{S^{\prime }}\exp \left({\sum }_{i<j}{w}_{ij}{s}_i^{\prime }{s}_j^{\prime }\right)}$
  • $P(V)={\sum }_H\frac{\exp \left({\sum }_{i<j}{w}_{ij}{s}_i{s}_j\right)}{{\sum }_{S^{\prime }}\exp \left({\sum }_{i<j}{w}_{ij}{s}_i^{\prime }{s}_j^{\prime }\right)}$

• Maximum Likelihood Training

  $$\log (P(V))=\log \left(\sum _H\exp \left(\sum _{i<j}{w}_{ij}{s}_i{s}_j\right)\right)-\log \left(\sum _{S_{\prime }}\exp \left(\sum _{i<j}{w}_{ij}{s}_i^{\prime }{s}_j^{\prime }\right)\right)$$

  $$\mathcal{L}=\frac{1}{N}\sum _{V\in \mathbf{V}}\log (P(V))$$
  $$\frac{d\mathcal{L}}{d{w}_{ij}}=\frac{1}{N}\sum _{V\in \mathbf{V}}\sum _{H}P(S\mid V)s_i{s}_j-\sum _{S!}P\left(S^{\prime }\right)s_i^{\prime }{s}_j^{\prime }$$

• ${\sum }_{H}P(S\mid V)s_i{s}_j\approx \frac{1}{K}{\sum }_{H\in {\mathbf{H}}_{simul}}{s}_i{S}_j$

• Computed as the average sampled hidden state with the visible bits fixed

• ${\sum }_{S^{\prime }}P\left(S^{\prime }\right)s_i^{\prime }{s}_j^{\prime }\approx \frac{1}{M}{\sum }_{S_i\in {\mathbf{S}}_{simul}}{s}_i^{\prime }{S}_j^{\prime }$

  • Computed as the average of sampled states when the network is running “freely”

Training

Step1

• For each training pattern $V_i$
  • Fix the visible units to $V_i$
  • Let the hidden neurons evolve from a random initial point to generate $H_i$
  • Generate $S_i=[V_i,H_i]$
• Repeat K times to generate synthetic training

S = { S 1 , 1 , S 1 , 2 , … , S 1 K , S 2 , 1 , … , S N , K } \mathbf{S}=\{S_{1,1}, S_{1,2}, \ldots, S_{1 K}, S_{2,1}, \ldots, S_{N, K}\} S={S1,1​,S1,2​,…,S1K​,S2,1​,…,SN,K​}

Step2

• Now unclamp the visible units and let the entire network evolve several times to generate

$${\mathbf{S}}_{simul}=S\_simul,1,S\_simul,2,\dots ,S\_simul,M$$

Gradients
$$\frac{d⟨\log (P(\mathbf{S}))⟩}{d{w}_{ij}}=\frac{1}{NK}\sum _{\mathbf{S}}{s}_i{s}_j-\frac{1}{M}\sum _{S_i\in {\mathbf{S}}_{\text{simul }}}{s}_i^{\prime }{s}_j^{\prime }$$

$$w_{ij}=w_{ij}-\eta \frac{d⟨\log (P(\mathbf{S}))⟩}{d{w}_{ij}}$$

• Gradients are computed as before, except that the first term is now computed over the expanded training data

Issues

• Training takes for ever
• Doesn’t really work for large problems
  • A small number of training instances over a small number of bits

Restricted Boltzmann Machines

• Partition visible and hidden units
  • Visible units ONLY talk to hidden units
  • Hidden units ONLY talk to visible units

Training

Step1

• For each sample
  • Anchor visible units
  • Sample from hidden units
  • No looping!!

Step2

• Now unclamp the visible units and let the entire network evolve several times to generate

$${\mathbf{S}}_{simul}=S\_simul,1,S\_simul,2,\dots ,S\_simul,M$$

• For each sample
  • Initialize $V_0$ (visible) to training instance value
  • Iteratively generate hidden and visible units
• Gradient

$$\frac{\partial \log p(v)}{\partial w_{ij}}=<v_i{h}_j{>}^0-<v_i{h}_j{>}^{\infty }$$

A Shortcut: Contrastive Divergence

• Recall: Raise the neighborhood of each target memory
• Sufficient to run one iteration to give a good estimate of the gradient

$$\frac{\partial \log p(v)}{\partial w_{ij}}=<v_i{h}_j{>}^0-<v_i{h}_j{>}^1$$