RBM深层理解

原创 2016年08月30日 10:32:08
要理解RBM,不得不首先理解什么是热平衡状态(thermal equilibrium),

 The thing that settles down is the probability distribution(P(v,h)) over configurations(给定可见层和隐层单元的一个指派). That's a difficult concept the first time you meet it, and so I'm going to giveyou an example. The probability distribution settles to a particular distribution called the Stationary Distribution. The stationary distribution is determined by the energy function of the system.(当然包括权重W了,且W是固定的) And, in fact,in the stationary distribution, the probability of any configuration is proportional to each of the minus its energy. A nice intuitive way to think about thermal equilibrium is to imagine a huge ensemble ofidentical systems that all have exactly the same energy function. So, imagine a very large number of stochastic Hopfield nets all with the same weights. Now, in that huge ensemble(全体), we can define the probability of configuration as the fraction of the systems that are in that configuration.

So, now we canunderstand what's happening as we approach thermal equilibrium.

We can start with any distribution we like

over all these identical systems. We could

make them all,be in the same

configuration.So, that's the distribution

with a property of one on one

configuration,and zero on everything

else. Or we could start them off, with an

equal number ofsystems in each possible

configuration.

So that's auniform distribution.

And then, we're going to keep applying our

stochastic update rule.

Which, in the case of a stochastic

Hopfield net would mean,

You pick a unit, and you look at its

energy gap.

And you make a random decision based on

that energy gap about whether to turn it

on orturn it off.

Then, you go and pick another unit, and so

on.

We keep applying that stochastic rule.

And after we've run systems stochastically

in this way,

We may eventually reach a situation where

the fraction of the systems in each

configuration remains constant.

We apply the update rule,

And the states of its units will keep

flipping between zero and one.

But, the fraction of systems in any

particular configuration doesn't change.

And that's because we have many, many more

systems than we have configurations.

In general, we're interested in reaching

equilibrium for systems where some

configurations have lower energy than

others.


如果理解了上面的概念,那试着看看下面的说法能否理解:

We want to maximize the product of the probabilities that the Boltzmann machine assigns to the binary vectors in the training set.
– This is equivalent to maximizing the sum of the
log probabilities that the Boltzmann machine
assigns to the training vectors.

It is also equivalent to maximizing the probability that we would obtain exactly the N training cases if we did the following
– Let the network settle to its stationary distribution N
different times with no external input.
– Sample the visible vector  once each time.

第一个概念很容易理解,第二个是什么意思呢,通过从stationary distribution抽样得到的visible vector和我们的 training data visible vector用于学习BM网络效果是一样的,也就是最后得到的网络是一致的


下面让我们看一个有趣的事


权重更新有两项,第一项固定V可见单元达到热平衡状态,第二项不固定v达到热平衡状态,为什么要减去第二项呢,因为第二项与V无关,即使没有V,si和sj同时为1的期望,不减掉的话就多学习了


An inefficient way to collect the statistics required for learning

Positive phase: Clamp a data
vector on the visible units and set
the hidden units to random
binary states.
– Update the hidden units one
at a time until the network
reaches thermal equilibrium
at a temperature of 1.
– Sample for every
connected pair of units.
– Repeat for all data vectors in
the training set and average

Negative phase: Set all the
units to random binary states.
– Update all the units one at
a time until the network
reaches thermal
equilibrium at a
temperature of 1.
– Sample for every
connected pair of units.
– Repeat many times (how
many?) and average to get
good estimates.


版权声明:本文为博主原创文章,未经博主允许不得转载。

相关文章推荐

RBM 算法理解

  • 2014-04-29 00:39
  • 775KB
  • 下载

RBM算法理解

  • 2015-04-06 09:43
  • 775KB
  • 下载

深度学习Deep learning:四十九(RNN-RBM简单理解)

前言:   本文主要是bengio的deep learning tutorial教程主页中最后一个sample:rnn-rbm in polyphonic music. 即用RNN-RBM来mode...

RBM的理解

前言    对RBM(Restricted Boltzmann Machine)的理解进行简单的总结。   主要涉及RBM的图结构,为什么场能E(v,h)要如此定义,优化的思路到底是怎...

rbm C++代码理解

代码链接:https://github.com/yusugomori/DeepLearning

神经网络的理解(MLP RBF RBM DBN DBM CNN 整理学习)

原文 http://lanbing510.info/2014/11/07/Neural-Network.html 开篇语 文章整理自向世明老师的PPT,围绕神经网络发展...

RBM简单理解-tornadomeet哥的原文

Deep learning:十九(RBM简单理解)     这篇博客主要用来简单介绍下RBM网络,因为deep learning中的一个重要网络结构DBN就可以由RBM网络叠加而成,所以对RB...

Java 中深层理解父类引用指向子类对象

从对象的内存角度来理解试试. 假设现在有一个父类Father,它里面的变量需要占用1M内存.有一个它的子类Son,它里面的变量需要占用0.5M内存. 现在通过代码来看看内存的分配情况:Father...

回溯与栈如何进行深层理解

回溯算法也叫试探法,它是一种系统地搜索问题的解的方法。回溯算法的基本思想是:从一条路往前走,能进则进,不能进则退回来,换一条路再试。  用回溯算法解决问题的一般步骤为:   一、定义一个解空间,它包含...
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:深度学习:神经网络中的前向传播和反向传播算法推导
举报原因:
原因补充:

(最多只允许输入30个字)