CMU 11-785 L20 Boltzmann machines 1

Training hopfield nets

Geometric approach

• $\mathbf{W}=\mathbf{Y}{\mathbf{Y}}^T-N_p\mathbf{I}$

• $\mathbf{E}(\mathbf{y})={\mathbf{y}}^T\mathbf{W}\mathbf{y}$

• Sine : ${\mathbf{y}}^T\left(\mathbf{Y}{\mathbf{Y}}^T-N_p\mathbf{I}\right)\mathbf{y}={\mathbf{y}}^T\mathbf{Y}{\mathbf{Y}}^T\mathbf{y}-N{N}_p$

• So W is identical to behavior with $\mathbf{W}=\mathbf{Y}{\mathbf{Y}}^T$

  • Energy landscape only differs by an additive constant
  • Have the same eigen vectors

• A pattern $y_p$ is stored if:

  • $\mathrm{sign}\left(\mathbf{W}{\mathbf{y}}_p\right)=\mathbf{y}\_p$ for all target patterns

• Training: Design $W$ such that this holds

• Simple solution: $y_p$ is an Eigenvector of $W$

Storing k orthogonal patterns

• Let $\mathbf{Y}=\left[\mathbf{y}\_1\mathbf{y}\_2\dots \mathbf{y}\_K\right]$
  • $\mathbf{W}=\mathbf{Y}\Lambda {\mathbf{Y}}^T$
  • ${\lambda }_1,...,{\lambda }_k$ are positive
  • for ${\lambda }_1={\lambda }_2={\lambda }_k=1$ this is exactly the Hebbian rule
• Any pattern $y$ can be written as
  • $\mathbf{y}=a_1{\mathbf{y}}_1+a_2{\mathbf{y}}_2+\cdots +a_N{\mathbf{y}}_N$
  • $\mathbf{W}\mathbf{y}=a_1{\mathbf{W}\mathbf{y}}_1+a_2{\mathbf{W}\mathbf{y}}_2+\cdots +a_N{\mathbf{W}\mathbf{y}}_N=y$
• All patterns are stable
  • Remembers everything
  • Completely useless network
• Even if we store fewer than $N$ patterns
  • Let $Y=\left[\mathbf{y}\_1\mathbf{y}\_2\dots \mathbf{y}\_K\mathbf{r}\_K+1\mathbf{r}\_K+2\dots \mathbf{r}\_N\right]$
  • $W=Y\Lambda Y^T$
  • $\mathbf{r}\_K+1\mathbf{r}\_K+2\dots \mathbf{r}\_N$ are orthogonal to ${\mathbf{y}}_1{\mathbf{y}}_2\dots {\mathbf{y}}_K$
  • ${\lambda }_1={\lambda }_2={\lambda }_k=1$
  • Problem arise because eigen values are all 1.0
    • Ensures stationarity of vectors in the subspace
    • All stored patterns are equally important

General (nonorthogonal) vectors

• w j i = ∑ p ∈ { p } y i p y j p w_{j i}=\sum_{p \in\{p\}} y_{i}^{p} y_{j}^{p} wji​=∑p∈{p}​yip​yjp​
• The maximum number of stationary patterns is actually exponential in $N$ (McElice and Posner, 84’)
• For a specific set of $K$ patterns, we can always build a network for which all $K$ patterns are stable provided $k\le N$
  • But this may come with many “parasitic” memories

Optimization

• Energy function
  • $E=-\frac{1}{2}{\mathbf{y}}^T\mathbf{W}\mathbf{y}-{\mathbf{b}}^T\mathbf{y}$
  • This must be maximally low for target patterns
  • Must be maximally high for all other patterns
    • So that they are unstable and evolve into one of the target patterns
• Estimate $W$ such that
  • $E$ is minimized for $y_1,...,y_P$
  • $E$ is maximized for all other $y$
• Minimize total energy of target patterns
  • $E(\mathbf{y})=-\frac{1}{2}{\mathbf{y}}^T\mathbf{W}\mathbf{y}\hat{\mathbf{W}}=\underset{\mathbf{W}}{\mathrm{argmin}}{\sum }_{\mathbf{y}\in {\mathbf{Y}}_P}E(\mathbf{y})$
  • However, might also pull all the neighborhood states down
• Maximize the total energy of all non-target patterns
  • $E(\mathbf{y})=-\frac{1}{2}{\mathbf{y}}^T\mathbf{W}\mathbf{y}$
  • $\hat{\mathbf{W}}=\underset{\mathbf{W}}{\mathrm{argmin}}{\sum }_{\mathbf{y}\in {\mathbf{Y}}_P}E(\mathbf{y})-{\sum }_{\mathbf{y}\notin {\mathbf{Y}}_P}E(\mathbf{y})$
• Simple gradient descent

  • $\mathbf{W}=\mathbf{w}+\eta \left({\sum }_{\mathbf{y}\in {\mathbf{Y}}_P}\mathbf{y}{\mathbf{y}}^T-{\sum }_{\mathbf{y}\notin {\mathbf{Y}}_P}\mathbf{y}{\mathbf{y}}^T\right)$

  • minimize the energy at target patterns

    • 

  • raise all non-target patterns

    • Do we need to raise everything?

Raise negative class

• Focus on raising the valleys

  • If you raise every valley, eventually they’ll all move up above the target patterns, and many will even vanish
  • 

• How do you identify the valleys for the current $W$?

  • 

  • Initialize the network randomly and let it evolve

  • It will settle in a valley

• Should we randomly sample valleys?

  • Are all valleys equally important?
  • Major requirement: memories must be stable
    • They must be broad valleys
  • 

• Solution: initialize the network at valid memories and let it evolve

  • It will settle in a valley
  • If this is not the target pattern, raise it

• What if there’s another target pattern downvalley

  • 

  • no need to raise the entire surface, or even every valley

    • Raise the neighborhood of each target memory

• 

Storing more than N patterns

• Visible neurons
  • The neurons that store the actual patterns of interest
• Hidden neurons
  • The neurons that only serve to increase the capacity but whose actual values are not important

• The maximum number of patterns the net can store is bounded by the width $N$ of the patterns…
• So lets pad the patterns with $K$ “don’t care” bits
  • The new width of the patterns is $N+K$
  • Now we can store $N+K$ patterns!
• Taking advantage of don’t care bits
  • Simple random setting of don’t care bits, and using the usual training and recall strategies for Hopfield nets should work
  • However, to exploit it properly, it helps to view the Hopfield net differently: as a probabilistic machine

A probabilistic interpretation

• For binary y the energy of a pattern is the analog of the negative log likelihood of a Boltzmann distribution
  • Minimizing energy maximizes log likelihood
  • $E(\mathbf{y})=-\frac{1}{2}{\mathbf{y}}^T\mathbf{W}\mathbf{y}P(\mathbf{y})=\mathrm{Cexp}(-E(\mathbf{y}))$

Boltzmann Distribution

• $E(\mathbf{y})=-\frac{1}{2}{\mathbf{y}}^T\mathbf{W}\mathbf{y}-{\mathbf{b}}^T\mathbf{y}$
• $P(\mathbf{y})=\mathrm{Cexp}\left(\frac{-E(\mathbf{y})}{kT}\right)$
• $C=\frac{1}{{\sum }_{\mathrm{y}}\exp \left(\frac{-E(\mathbf{y})}{kT}\right)}$
• $k$ is the Boltzmann constant, $T$ is the temperature of the system
• Optimizing $W$
  • $E(\mathbf{y})=-\frac{1}{2}{\mathbf{y}}^T\mathbf{W}\mathbf{y}\hat{\mathbf{W}}=\underset{\mathbf{W}}{\mathrm{argmin}}{\sum }_{\mathbf{y}\in {\mathbf{Y}}_P}E(\mathbf{y})-{\sum }_{\mathbf{y}\notin {\mathbf{Y}}_P}E(\mathbf{y})$
  • Simple gradient descent
  • $\mathbf{W}=\mathbf{W}+\eta \left({\sum }_{\mathbf{y}\in {\mathbf{Y}}_P}{\alpha }_{\mathbf{y}}\mathbf{y}{\mathbf{y}}^T-{\sum }_{\mathbf{y}\notin {\mathbf{Y}}_P}\beta (E(\mathbf{y}))\mathbf{y}{\mathbf{y}}^T\right)$
  • ${\alpha }_y$ more importance to more frequently presented memories
  • $\beta (E(y))$ more importance to more attractive spurious memories
  • Looks like an expectation
  • $\mathbf{W}=\mathbf{W}+\eta \left(E_{\mathbf{y}\sim {\mathbf{Y}}_P}\mathbf{y}{\mathbf{y}}^T-E_{\mathbf{y}\sim Y}\mathbf{y}{\mathbf{y}}^T\right)$
• The behavior of the Hopfield net is analogous to annealed dynamics of a spin glass characterized by a Boltzmann distribution