要理解RBM,不得不首先理解什么是热平衡状态(thermal equilibrium),

 The thing that settles down is the probability distribution(P(v,h)) over configurations(给定可见层和隐层单元的一个指派). That's a difficult concept the first time you meet it, and so I'm going to giveyou an example. The probability distribution settles to a particular distribution called the Stationary Distribution. The stationary distribution is determined by the energy function of the system.(当然包括权重W了,且W是固定的) And, in fact,in the stationary distribution, the probability of any configuration is proportional to each of the minus its energy. A nice intuitive way to think about thermal equilibrium is to imagine a huge ensemble ofidentical systems that all have exactly the same energy function. So, imagine a very large number of stochastic Hopfield nets all with the same weights. Now, in that huge ensemble(全体), we can define the probability of configuration as the fraction of the systems that are in that configuration.

So, now we canunderstand what's happening as we approach thermal equilibrium.

We can start with any distribution we like

over all these identical systems. We could

make them all,be in the same

configuration.So, that's the distribution

with a property of one on one

configuration,and zero on everything

else. Or we could start them off, with an

equal number ofsystems in each possible


So that's auniform distribution.

And then, we're going to keep applying our

stochastic update rule.

Which, in the case of a stochastic

Hopfield net would mean,

You pick a unit, and you look at its

energy gap.

And you make a random decision based on

that energy gap about whether to turn it

on orturn it off.

Then, you go and pick another unit, and so


We keep applying that stochastic rule.

And after we've run systems stochastically

in this way,

We may eventually reach a situation where

the fraction of the systems in each

configuration remains constant.

We apply the update rule,

And the states of its units will keep

flipping between zero and one.

But, the fraction of systems in any

particular configuration doesn't change.

And that's because we have many, many more

systems than we have configurations.

In general, we're interested in reaching

equilibrium for systems where some

configurations have lower energy than



We want to maximize the product of the probabilities that the Boltzmann machine assigns to the binary vectors in the training set.
– This is equivalent to maximizing the sum of the
log probabilities that the Boltzmann machine
assigns to the training vectors.

It is also equivalent to maximizing the probability that we would obtain exactly the N training cases if we did the following
– Let the network settle to its stationary distribution N
different times with no external input.
– Sample the visible vector  once each time.

第一个概念很容易理解,第二个是什么意思呢,通过从stationary distribution抽样得到的visible vector和我们的 training data visible vector用于学习BM网络效果是一样的,也就是最后得到的网络是一致的



An inefficient way to collect the statistics required for learning

Positive phase: Clamp a data
vector on the visible units and set
the hidden units to random
binary states.
– Update the hidden units one
at a time until the network
reaches thermal equilibrium
at a temperature of 1.
– Sample for every
connected pair of units.
– Repeat for all data vectors in
the training set and average

Negative phase: Set all the
units to random binary states.
– Update all the units one at
a time until the network
reaches thermal
equilibrium at a
temperature of 1.
– Sample for every
connected pair of units.
– Repeat many times (how
many?) and average to get
good estimates.

个人分类: 深度学习
想对作者说点什么? 我来说一句