(本文为本人学习工作总结,如有雷同,不胜荣幸。可联系本人立即修改或者删除)
RNN(循环网络)
问题:什么是RNN?RNN有什么应用?怎么用?
解答:
RNN
概念
循环卷积网络(RNN,Recurrent Neural Networks),将数据关联单个NN,RNN在x(t)时刻得出状态描述S(t),由此输出Y(t);在x(t+1)时刻,产生S(t+1),而Y(t+1)由S(t)和S(t+1)产生。
缺陷:梯度消失(e.g:state传递下去会出现0.1^n,趋向于0),梯度爆炸(e.g:1.1^n,趋向于∞);解决:使用LSTM Cell(Long-Short Term Memory Cell),结构:在RNN 的cell添加Input Gate,output Gate,forget Gate.这些Gate是一些参数,Input记住memory,output输出memory,forget忘记memory。
论文
参考部分
总结
应用
用于序列化数据
1. RNN描述照片
2. 写程序
3. 创作歌曲
4. 创作诗歌论文
代码实现
1.classfication
TensorFlow实现:
RNN_classfication.py
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
import numpy as np
import matplotlib.pyplot as plt
#1.3.0-0
print(tf.__version__)
print(np.__version__)
#设置种子数
tf.set_random_seed(1)
np.random.seed(1)
#超参数(自定义的参数)
BATCH_SIZE = 64
TIME_STEP = 28
INPUT_SIZE =28
LR = 0.01
#数据
mnist = input_data.read_data_sets('./mnist', one_hot=True)
test_x = mnist.test.images[:2000]
test_y = mnist.test.labels[:2000]
#输出测试
print(mnist.train.images.shape)
print(mnist.train.labels.shape)
plt.imshow(mnist.train.images[0].reshape((28,28)),cmap='gray')
plt.title('%i' % np.argmax(mnist.train.labels[0]))
plt.show()
#tensorflow placeholders
##def placeholder(dtype, shape=None, name=None):shape(batch , 28*28=784)
tf_x = tf.placeholder(tf.float32,[None, TIME_STEP * INPUT_SIZE])
##def reshape(tensor, shape, name):shape(batch, height, width, channel)
image = tf.reshape(tf_x, [-1, TIME_STEP, INPUT_SIZE])
tf_y = tf.placeholder(tf.int32, [None, 10])
#RNN结构
rnn_cell = tf.contrib.rnn.BasicLSTMCell(num_units=64)
##def dynamic_rnn(cell, inputs, sequence_length=None, initial_state=None,
## dtype=None, parallel_iterations=None, swap_memory=False,
## time_major=False, scope=None):
outputs, (h_c,h_n) = tf.nn.dynamic_rnn(
rnn_cell, #An instance of RNNCell
image, #inputs
initial_state=None, #初始化的隐藏状态
dtype=tf.float32, #隐藏状态为None时,dtype必须设置
time_major=False, #False: (batch, time step, input); True: (time step, batch, input)
)
print(outputs)
output = tf.layers.dense(outputs[:, -1, :], 10)
##softmax算法,
loss = tf.losses.softmax_cross_entropy(onehot_labels=tf_y, logits=output)
##adam(学习率)
train_op = tf.train.AdamOptimizer(LR).minimize(loss)
##def accuracy(labels, predictions, weights=None, metrics_collections=None,
## updates_collections=None, name=None): return (acc, update_op), and create 2 local variables
###def argmax(input,axis=None,name=None,dimension=None,output_type=dtypes.int64):
accuracy = tf.metrics.accuracy(
labels=tf.argmax(tf_y,axis=1), predictions=tf.argmax(output,axis=1),)[1]
sess = tf.Session()
init_op = tf.group(tf.global_variables_initializer(),tf.local_variables_initializer())
# 在图中初始化参数
sess.run(init_op)
#训练
for step in range(1200):
##返回值,return self._images[start:end], self._labels[start:end]
b_x, b_y = mnist.train.next_batch(BATCH_SIZE)
##run(self, fetches, feed_dict=None, options=None, run_metadata=None) :Runs operations and evaluates tensors in `fetches`.
_, loss_ = sess.run([train_op,loss], {tf_x: b_x, tf_y: b_y})
if step % 50 == 0:
accuracy_ =sess.run(accuracy, {tf_x:test_x, tf_y:test_y})
print('train loss: %.4f' % loss_, '| test accuracy: %.2f' % accuracy_)
#输出10个预测值
test_output = sess.run(output, {tf_x:test_x[:10]})
pred_y = np.argmax(test_output, 1)
print(pred_y, 'prediction number')
print(np.argmax(test_y[:10], 1), 'real number')
注意:程序运行时正在下mnist手写训练集合,会卡住控制台,最好的办法就是提前下载好训练集放到对应的目录.
输出:
/usr/local/anaconda3/bin/python3.5 /usr/local/IdeaProjects/MyCSDN/Tensorflow_test/RNN_classification.py
1.3.0
1.11.1
Extracting ./mnist/train-images-idx3-ubyte.gz
Extracting ./mnist/train-labels-idx1-ubyte.gz
Extracting ./mnist/t10k-images-idx3-ubyte.gz
Extracting ./mnist/t10k-labels-idx1-ubyte.gz
(55000, 784)
(55000, 10)
(process:7948): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
GLib-GIO-Message: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications.
(python3.5:7948): GConf-WARNING **: Client failed to connect to the D-BUS daemon:
Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_register_object: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_register_object: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_get_unique_name: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_register_object: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_register_object: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
(python3.5:7948): GLib-GIO-CRITICAL **: g_dbus_connection_get_unique_name: assertion 'G_IS_DBUS_CONNECTION (connection)' failed
Qt: Session management error: None of the authentication protocols specified are supported
Tensor("rnn/transpose:0", shape=(?, 28, 64), dtype=float32)
2017-11-12 21:15:47.124601: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-12 21:15:47.124622: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-12 21:15:47.124626: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-11-12 21:15:47.124630: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-12 21:15:47.124634: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
train loss: 2.3040 | test accuracy: 0.13
train loss: 0.8905 | test accuracy: 0.41
train loss: 0.6223 | test accuracy: 0.52
train loss: 0.4808 | test accuracy: 0.60
train loss: 0.4486 | test accuracy: 0.65
train loss: 0.1816 | test accuracy: 0.70
train loss: 0.4413 | test accuracy: 0.73
train loss: 0.0996 | test accuracy: 0.75
train loss: 0.1570 | test accuracy: 0.77
train loss: 0.3565 | test accuracy: 0.78
train loss: 0.0979 | test accuracy: 0.80
train loss: 0.2561 | test accuracy: 0.81
train loss: 0.1261 | test accuracy: 0.82
train loss: 0.1363 | test accuracy: 0.83
train loss: 0.1119 | test accuracy: 0.84
train loss: 0.2960 | test accuracy: 0.85
train loss: 0.0805 | test accuracy: 0.85
train loss: 0.2942 | test accuracy: 0.86
train loss: 0.2955 | test accuracy: 0.86
train loss: 0.0553 | test accuracy: 0.87
train loss: 0.0917 | test accuracy: 0.87
train loss: 0.1553 | test accuracy: 0.87
train loss: 0.2084 | test accuracy: 0.88
train loss: 0.1777 | test accuracy: 0.88
[7 2 1 0 4 1 4 9 5 9] prediction number
[7 2 1 0 4 1 4 9 5 9] real number
Process finished with exit code 0
特别感谢莫烦大神的教程
参考
- Towards End-to-End Speech Recognition
with Recurrent Neural Networks
2.格雷夫斯,亚历克斯 “用循环神经网络生成序列
3.莫烦大神的教程
4.github llSourcell/LSTM_Networks