为了分析一个神经元的反向传播过程,对一个2×2的感受野卷积一次,通过学习,让它认识这个感受野为1。
- 感受野的输入信号
为
。
- padding='VALID',不然会卷积4次。
- 优化器:GradientDescentOptimizer。
- 激活函数:sigmoid。
- 学习率恒为0.2。
![](https://i-blog.csdnimg.cn/blog_migrate/10791ad8db5a665c7bb5db5342737c36.jpeg)
神经元如图1所示, 那么:
- 损失函数:
- 损失函数对权重的偏导数:
- 损失函数对偏置的偏导数:
正向传播和反向传播
在一次训练时,滤波器被初始化为:
,偏置
被初始化为
。第一次迭代:
正向传播
反向传播
1. Loss对权重和偏置的偏导和通过tf.gradients(loss,filter)计算的一样(只有最后1位数有较小差异):
1.1 程序输出的梯度:--filter gradient: [[-0.02498191 -0.04996381 -0.07494572 -0.09992762]] bias gradient: [[-0.24981906]]
1.2 手动计算的梯度:
2. 更新权重和偏置:
2.1 程序更新的权重和偏置:--filter: [[0.00398535 0.01192442 0.00282736 0.03200994]] bias: [0.04996381]
2.2 手动更新的权重和偏置:
迭代
反向传播一次,w和b就更新一次。多次这样的更新后,wx+b就越来越接近1,这就是训练的目的。
相关程序
程序输出:
--trainable variables: [<tf.Variable 'filter:0' shape=(2, 2, 1, 1) dtype=float64_ref>, <tf.Variable 'bias:0' shape=(1,) dtype=float64_ref>]
--filter: [[-0.00101103 0.00193166 -0.01216178 0.01202441]] bias: [0.]
--y1: [[[[0.00144646]]]] y2: [[[[0.50036161]]]] loss: 0.2496385164748249
--filter gradient: [[-0.02498191 -0.04996381 -0.07494572 -0.09992762]] bias gradient: [[-0.24981906]]
--filter: [[0.00398535 0.01192442 0.00282736 0.03200994]] bias: [0.04996381]
--y1: [[[[0.0164356]]]] y2: [[[[0.51659376]]]] loss: 0.23368159536065822
--filter gradient: [[-0.02414369 -0.04828738 -0.07243107 -0.09657476]] bias gradient: [[-0.24143691]]
--filter: [[0.00881409 0.02158189 0.01731358 0.05132489]] bias: [0.0982512]
--y1: [[[[0.03092182]]]] y2: [[[[0.53224842]]]] loss: 0.21879153616090155
--filter gradient: [[-0.02329029 -0.04658058 -0.06987087 -0.09316116]] bias gradient: [[-0.2329029]]
代码:
import tensorflow as tf
import numpy as np
def net(input):
global filter, bias, y1, y2
init_random = tf.random_normal_initializer(mean=0.0, stddev=0.01, seed=None, dtype=tf.float64)
filter = tf.get_variable('filter', shape=[2,2,1,1], initializer=init_random, dtype=tf.float64)
bias = tf.Variable([0], dtype=tf.float64, name='bias')
y1 = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='VALID')
y2 = tf.nn.sigmoid(y1 + bias)
return y2
def display(sess):
#print '--it:%2d' % it,'loss:',loss.eval({input:data},sess)
print
print"--filter:",filter.eval(sess).reshape(1,4)," bias:",bias.eval(sess)
print"--y1:",y1.eval({input:data},sess)," y2:",y2.eval({input:data},sess),"loss:",loss.eval({input:data},sess)
print"--filter gradient:",tf.gradients(loss,filter)[0].eval({input:data},sess).reshape(1,4),\
" bias gradient:",tf.gradients(loss,bias)[0].eval({input:data},sess).reshape(1,1)
data = np.array([[0.1,0.2],[0.3,0.4]])
data = np.reshape(data,(1,2,2,1))
input = tf.placeholder(tf.float64, [1,2,2,1])
predict = net(input)
loss = tf.reduce_mean(tf.square(1-predict))
step = tf.Variable(0, trainable=False)
rate = tf.train.exponential_decay(0.2, step, 1, 1)
#optimizer = tf.train.AdadeltaOptimizer(rate)
#optimizer = tf.train.AdagradOptimizer(rate)
#optimizer = tf.train.AdamOptimizer(rate)
#optimizer = tf.train.FtrlOptimizer(rate)
optimizer = tf.train.GradientDescentOptimizer(rate)
#optimizer = tf.train.MomentumOptimizer(rate)
#optimizer = tf.train.RMSPropOptimizer(rate)
train = optimizer.minimize(loss, global_step=step)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
print"--trainable variables:", tf.trainable_variables()
for it in range(3):
display(sess)
train.run({input:data},sess)
更多
当padding='SAME'时会有4个卷积结果,变成4分类问题:
import tensorflow as tf
import numpy as np
def net(input):
global filter, bias
init_random = tf.random_normal_initializer(mean=0.0, stddev=0.01, seed=None, dtype=tf.float32)
filter = tf.get_variable('filter', shape=[2,2,1,1], initializer=init_random)
bias = tf.Variable([0], dtype=tf.float32, name='bias')
out = tf.nn.conv2d(input, filter, strides=[1, 1, 1, 1], padding='SAME')
out = tf.nn.sigmoid(out + bias)
return out
def display(sess):
print"--trainable variables:", tf.trainable_variables()
print"--filter:", filter.eval(sess).reshape(1,4)
print"--bias:", bias.eval(sess)
data = np.array([[1,2],[3,4]])
data = np.reshape(data,(1,2,2,1))
input = tf.placeholder(tf.float32, [1,2,2,1])
predict = net(input)
GT = tf.constant([0,1,0,0], shape=(1,2,2,1), dtype=tf.float32)
loss = tf.reduce_mean(tf.square(GT-predict))
step = tf.Variable(0, trainable=False)
rate = tf.train.exponential_decay(0.15, step, 1, 0.9999)
optimizer = tf.train.AdamOptimizer(rate)
train = optimizer.minimize(loss, global_step=step)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
display(sess)
for it in range(3):
train.run({input:data},sess)
print '--it:%2d' % it,'loss:',loss.eval({input:data},sess)
display(sess)