RBM (受限玻尔兹曼机)

最新推荐文章于 2020-09-29 11:03:53 发布

schwt

最新推荐文章于 2020-09-29 11:03:53 发布

阅读量605

点赞数

分类专栏： ML整理笔记

本文链接：https://blog.csdn.net/schwt/article/details/49466879

版权

ML整理笔记专栏收录该内容

12 篇文章 0 订阅

订阅专栏

1. 玻尔兹曼分布：

p (E) \sim e - E / k T

$p(E) \thicksim e^{-E/kT}$

2. RBM

两层：隐层和可视层， $\mathbf v$ , $\mathbf h$

v i \in {0, 1}, h j \in {0, 1}

$v_i \in \{0, 1\}, \ \ \ h_j \in \{0, 1\}$

能量假设：

$E (v, h; θ) = - b \cdot v - c \cdot h - v T W h θ = {b, c, W}$ $E (\mathbf v, \mathbf h; \theta) = - \mathbf b \cdot \mathbf v - \mathbf c \cdot \mathbf h - \mathbf v^T W \mathbf h \\ \theta = \{\mathbf b,\ \mathbf c,\ W\}$
概率分布：

$p (v, h; θ) = 1 Z e - E (v, h; θ) Z (θ) = \sum v, h e - E (v, h; θ)$ $p(v, h; \theta) = \frac 1 Z e ^{-E(v,h; \ \theta)} \\ Z(\theta) = \sum_{v,h} e^{-E(v,h; \theta)}$
条件概率：

$p (v | h; θ) = e - E ( v , h ) \sum v e - E ( v , h ) p (h | v; θ) = e - E ( v , h ) \sum h e - E ( v , h ) p (v i = 1 | h; θ) = σ (b i + \sum j W i j h j) p (h j = 1 | v; θ) = σ (c j + \sum i W i j v i)$ $p(v | h;\ \theta) = \frac{e^{-E(v,h)}}{\sum_ve^{-E(v,h)}} \\ p(h | v;\ \theta) = \frac{e^{-E(v,h)}}{\sum_he^{-E(v,h)}} \\ p(v_i = 1 \ | \ h; \theta) = \sigma(b_i + \sum_j W_{ij} h_j) \\ p(h_j = 1 \ | \ v; \theta) = \sigma(c_j + \sum_i W_{ij} v_i)$
全概率：

$p (v) = \sum h p (v, h) = \sum h e - E ( v , h ) \sum v , h e - E ( v , h )$ $p(v) = \sum_h p(v,h) = \frac{\sum_h e^{-E(v,h)}}{\sum_{v,h} e^{-E(v,h)}}$

3. 优化

极大化似然函数：

$ (θ | v) = ln p (v; θ) = ln \sum h e - E (v, h) - ln \sum v, h e - E (v, h)$ $\mathcal L(\theta\ |\ v) = \ln p(v;\ \theta) = \ln\sum_he^{-E(v,h)} - \ln\sum_{v,h} e^{-E(v,h)}$
梯度：

$\partial L \partial θ = E p (h | v) [- \partial E ( v , h ) \partial θ] - E p (v, h) [- \partial E ( v , h ) \partial θ] \partial E ( v , h ) \partial W i j = - v i h j, \partial E ( v , h ) \partial b i = - v i, \partial E ( v , h ) \partial c j = - h j$ $\frac{\partial L}{\partial \theta} = \mathrm E_{p(h|v)}[-\frac{\partial E(v,h)}{\partial\theta}] - \mathrm E_{p(v,h)}[-\frac{\partial E(v,h)}{\partial \theta}] \\ \frac{\partial E(v,h)}{\partial W_{ij}} = - v_i h_j, \\ \frac{\partial E(v,h)}{\partial b_i} = - v_i, \\ \frac{\partial E(v,h)}{\partial c_j} = - h_j$

4. 其他能量模型

1) Gaussian-Bernoulli RBM：

能量定义：

$E (v, h; θ) = \sum i ( v i - b i ) 2 2 σ 2 i - \sum j c j h j - \sum i j W i j v i σ i h j θ = {b, σ, c, W}$ $E(v,h; \theta) = \sum_i \frac{(v_i - b_i)^2}{2\sigma_i^2} - \sum_j c_j h_j - \sum_{ij} W_{ij}\frac{v_i}{\sigma_i}h_j \\ \theta = \{b,\ \sigma,\ c,\ W \} \qquad\qquad \qquad\qquad$
条件概率：

$p (v i = x | h; θ) =  (b i + σ i \sum j W i j h j, σ i) p (h j = 1 | v; θ) = σ (c j + \sum i W i j v i σ i)$ $p(v_i = x \ | \ h;\ \theta) = \mathcal N(b_i + \sigma_i \sum_j W_{ij}h_j,\ \sigma_i) \\ p(h_j = 1 \ | \ v;\ \theta) = \sigma(c_j + \sum_i W_{ij}\frac {v_i}{\sigma_i}) \ \ \ \ \$

2) extended energy

能量定义
$E (v, y, h) = - \sum b i v i - \sum c j h j - \sum W i j v i h j - \sum d k y k - \sum U j k h j y k θ = {b, c, W, d, U}$ $E(v,\ y,\ h) = -\sum b_i v_i - \sum c_j h_j - \sum W_{ij} v_i h_j - \sum d_k y_k - \sum U_{jk} h_j y_k \\ \theta = \{b,\ c,\ W,\ d,\ U\} \qquad\qquad\qquad$
条件概率
$p (v i = 1 | h) = σ (b i + \sum j W i j h j) p (h j = 1 | x, y) = σ (c j + \sum i W i j x i + \sum k U j k y k) p (y k = 1 | h) = exp ( d k + \sum j U j k h j ) \sum k exp ( d k + \sum k U j k h j )$ $p(v_i = 1 | h) = \sigma(b_i + \sum_j W_{ij}h_j) \qquad\qquad\\ p(h_j = 1| x,\ y) = \sigma(c_j + \sum_iW_{ij}x_i + \sum_kU_{jk}y_k) \\ p(y_k = 1 | h) = \frac{\exp(d_k + \sum_j U_{jk}h_j)}{\sum_k \exp(d_k + \sum_kU_jk h_j)}$

5. 附录

1. 玻尔兹曼分布的最大熵推导

封闭系统能量守恒，总能量 $\mathcal E$ 。共有 $N$ 个状态，每个状态 $i$ 的能量 $E_i$ ，对应概率 $p_i$
则有约束条件：

\sum i p i = 1 \sum i p i E i =  / N \equiv E ¯

$\sum_ip_i = 1 \\ \sum_i p_i E_i = \mathcal E/N \equiv \bar E$
最大化信息熵：

H [p] = - \sum i p i ln p i

$H[p] = -\sum_ip_i\ln p_i$
等效于最大化下面的拉格朗日量：

 [p] = H [p] + α (1 - \sum i p i) + β (E ¯ - \sum i p i E i)

$\mathcal L[p] = H[p] + \alpha (1 - \sum_ip_i)+ \beta(\bar E - \sum_ip_iE_i)$
即得能量的概率分布：

p (E i) \propto e - β E i

$p(E_i) \propto e^{-\beta E_i}$

2. RBM 条件概率推导

p (v i = 1 | h) = \sum v k \neq i p ( v i = 1 , v k , h ) \sum v p ( v , h ) = \sum v k \neq i exp [ ( b i v i + \sum j W i j v i b j ) v i = 1 + \sum k \neq i b k v k + \sum j c j h j + \sum k \neq i , j W k j v k h j ] \sum v i , v k \neq i exp [ ( b i v i + \sum j W i j v i b j ) + \sum k \neq i b k v k + \sum j c j h j + \sum k \neq i , j W k j v k h j ] ] = exp [ ( b i v i + \sum j W i j v i b j ) v i = 1 ] \cdot \sum v k \neq i exp [ \sum k \neq i b k v k + \sum j c j h j + \sum k \neq i , j W k j v k h j ] \sum v i exp [ ( b i v i + \sum j W i j v i b j ) ] \cdot \sum v k \neq i exp [ \sum k \neq i b k v k + \sum j c j h j + \sum k \neq i , j W k j v k h j ] = exp [ ( b i v i + \sum j W i j v i b j ) v i = 1 ] \sum v i exp [ ( b i v i + \sum j W i j v i b j ) ] = 1 1 + exp [ - b i - \sum j W i j b j ] . (v i \in {0, 1})

$p(v_i=1 |h) = \frac{\sum_{v_k\ne i}p(v_i=1,v_k,h)}{\sum_vp(v,h)} \\ =\frac{\sum_{v_k\ne i}\exp[(b_iv_i+\sum_j W_{ij}v_ib_j)_{v_i=1}+\sum_{k\ne i}b_kv_k + \sum_j c_jh_j + \sum_{k\ne i,j}W_{kj}v_kh_j]} {\sum_{v_i,v_{k\ne i}} \exp[(b_iv_i+\sum_j W_{ij}v_ib_j)+\sum_{k\ne i}b_kv_k + \sum_j c_jh_j + \sum_{k\ne i,j}W_{kj}v_kh_j]]} \\ = \frac{\exp[(b_iv_i+\sum_j W_{ij}v_ib_j)_{v_i=1}]\cdot\sum_{v_k\ne i}\exp[\sum_{k\ne i}b_kv_k + \sum_j c_jh_j + \sum_{k\ne i,j}W_{kj}v_kh_j]} {\sum_{v_i} \exp[(b_iv_i+\sum_j W_{ij}v_ib_j)] \cdot \sum_{v_{k\ne i}}\exp[\sum_{k\ne i}b_kv_k + \sum_j c_jh_j + \sum_{k\ne i,j}W_{kj}v_kh_j]} \\ = \frac{\exp[(b_iv_i+\sum_j W_{ij}v_ib_j)_{v_i=1}]} {\sum_{v_i} \exp[(b_iv_i+\sum_j W_{ij}v_ib_j)]} \\ = \frac{1}{1 + \exp[-b_i - \sum_jW_{ij}b_j]}. \qquad (v_i\in \{0,1\})$