TensorFlow教程（十）：反向传播

TensorFlow的一个优势是它可以维护操作状态和基于反向传播自动更新模型变量。

 

TensorFlow通过声明优化函数（optimization function）来实现，一旦声明好优化函数，TensorFlow将通过它在计算图中解决反向传播的项。当传入数据，最小化损失函数，TensorFlow会在计算图中根据状态相应的调节变量。

 

1、回归算法。均值为1，标准差为0.1的正态分布中抽样随机数，然后乘以变量A，损失函数为L2正则损失函数。理论上，A的最优值是10，因为生成的样本数据均值为1。

#-*- coding:utf-8 -*-

import tensorflow as tf
import numpy as np

sess = tf.Session()

x_vals = np.random.normal(1, 0.1, 100)
y_vals = np.repeat(10.0, 100)
x_data = tf.placeholder(shape=[1], dtype=tf.float32)
y_target = tf.placeholder(shape=[1], dtype=tf.float32)
A = tf.Variable(tf.random_normal(shape=[1]))

my_output = tf.multiply(x_data, A)

loss = tf.square(my_output - y_target)

init = tf.global_variables_initializer()
sess.run(init)

#declare opt
my_opt = tf.train.GradientDescentOptimizer(learning_rate=0.02)
train_step = my_opt.minimize(loss)

#training
for i in range(100):
    rand_index = np.random.choice(100)
    rand_x = [x_vals[rand_index]]
    rand_y = [y_vals[rand_index]]
    sess.run(train_step, feed_dict={x_data:rand_x, y_target:rand_y})
    if (i+1)%25 == 0:
        print('Step #' + str(i + 1) + 'A = ' + str(sess.run(A)))
        print('Loss = ' + str(sess.run(loss, feed_dict={x_data:rand_x, y_target:rand_y})))


#Step #25A = [6.326008]
#Loss = [7.8349257]
#Step #50A = [8.514299]
#Loss = [2.130918]
#Step #75A = [9.472592]
#Loss = [1.8418382]
#Step #100A = [9.754903]
#Loss = [0.6592515]

2、二值分类算法。从两个正态分布（N(-1,1)和N(3,1)）生成100个数。所有从正态分布N(-1,1)生成的数据标为目标类0；从正态分布N(3,1)生成的数据标为类1，模型算法是sigmoid(x+A)，其中，A是要拟合的变量，理论上A=-1。假设，两个正太分布的均值分别是m1和m2，则达到的A的取值时，它们通过-(m1+m2)/2转换成到0等距的值。

#-*- coding:utf-8 -*-

import tensorflow as tf
import numpy as np

sess = tf.Session()

x_vals = np.concatenate((np.random.normal(-1, 1, 50), np.random.normal(3, 1, 50)))
y_vals = np.concatenate((np.repeat(0.0, 50), np.repeat(1.0, 50)))
x_data = tf.placeholder(shape=[1], dtype=tf.float32)
y_target = tf.placeholder(shape=[1], dtype=tf.float32)
A = tf.Variable(tf.random_normal(mean=10, shape=[1]))

my_output = tf.add(x_data, A)

my_output_expanded = tf.expand_dims(my_output, 0)
y_target_expanded = tf.expand_dims(y_target, 0)

init = tf.global_variables_initializer()
sess.run(init)

xentropy = tf.nn.sigmoid_cross_entropy_with_logits(logits=my_output_expanded, labels=y_target_expanded)

my_opt = tf.train.GradientDescentOptimizer(0.05)
train_step = my_opt.minimize(xentropy)

for i in range(1400):
    rand_index = np.random.choice(100)
    rand_x = [x_vals[rand_index]]
    rand_y = [y_vals[rand_index]]
    sess.run(train_step, feed_dict={x_data:rand_x, y_target:rand_y})
    if (i+1) % 200 == 0:
        print('Step #' + str(i+1) + 'A = '+ str(sess.run(A)))
        print('Loss = ' + str(sess.run(xentropy, feed_dict={x_data:rand_x, y_target:rand_y})))


#Step #200A = [3.1671317]
#Loss = [[1.5945358]]
#Step #400A = [0.42701474]
#Loss = [[0.25861406]]
#Step #600A = [-0.7353596]
#Loss = [[0.30368462]]
#Step #800A = [-1.0829395]
#Loss = [[0.1167491]]
#Step #1000A = [-1.1390519]
#Loss = [[0.8808856]]
#Step #1200A = [-1.0290167]
#Loss = [[0.2420652]]
#Step #1400A = [-1.1319011]
#Loss = [[0.23258743]]

==============================================================

学习率比较	优缺点及应用
大学习率	收敛慢，但是结果精确，适用于算法稳定的时候
小学习率	收敛快，但是结果不精确，适用于算法收敛太慢的时候