4.2 Dropout+Optimizer

学习内容：

• 防止过拟合的常用方法：
  1.增加数据集
  2.正则化方法：$c=c_0+\frac{\lambda }{2n}\sum w^2$
  3.dropout方法：指定层的神经元以一定的比例工作
• 常用优化器
  1.GradientDescentOptimizer
  2.AdadeltaOptimizer
  3.AdagradOptimizer
  4.MomentumOptimizer
  5.AdamOptimizer
  6.FtrlOptimizer
  7.ProximalGradientDescentOptimizer
  8.ProximalAdagradOptimizer
  9.AdagradDAOptimizer
  10.RMSPropOptimizer

代码：

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("C:/Users/WangT/Desktop/MNIST_data",one_hot=True)

batch_size = 100
n_batch = mnist.train.num_examples // batch_size

x = tf.placeholder(tf.float32,[None,784])
y = tf.placeholder(tf.float32,[None,10])
keep_prob = tf.placeholder(tf.float32)#为后续的dropout中的参数占空

W1 = tf.Variable(tf.truncated_normal([784,2000],stddev=0.1))
#tf.truncated_normal(shape, mean, stddev) :shape表示生成张量的维度，mean是均值，stddev是标准差。这个函数产生正太分布，均值和标准差自己设定，通常利用这个函数来初始化w
b1 = tf.Variable(tf.zeros([2000])+0.1)
L1 = tf.nn.tanh(tf.matmul(x,W1)+b1)
L1_drop = tf.nn.dropout(L1,keep_prob)
#利用dropout,对L1的计算，在前向传播的时候，让第二层的神经元的激活值以一定的概率keep_prob停止工作

W2 = tf.Variable(tf.truncated_normal([2000,1000],stddev=0.1))
b2 = tf.Variable(tf.zeros([1000])+0.1)
L2 = tf.nn.tanh(tf.matmul(L1_drop,W2)+b2)
L2_drop = tf.nn.dropout(L2,keep_prob)

W3 = tf.Variable(tf.truncated_normal([1000,1000],stddev=0.1))
b3 = tf.Variable(tf.zeros([1000])+0.1)
L3 = tf.nn.tanh(tf.matmul(L2_drop,W3)+b3)
L3_drop = tf.nn.dropout(L3,keep_prob)

W4 = tf.Variable(tf.truncated_normal([1000,10],stddev=0.1))
b4 = tf.Variable(tf.zeros([10])+0.1)
prediction = tf.nn.softmax(tf.matmul(L3_drop,W4)+b4)
#得到预测结果
# loss = tf.reduce_mean(tf.square(y - prediction))
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y,logits=prediction))
#损失函数，评估模型好坏，tf.square是平方，tf.reduce_mean是取平均值
#train_step = tf.train.GradientDescentOptimizer(0.2).minimize(loss)
train_step = tf.train.AdamOptimizer(1e-3).minimize(loss)
#利用Adaoptimizer（）优化器，设置初始学习速率

init = tf.global_variables_initializer()
#添加一个操作来初始化变量
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(prediction,1))
#tf.equal()比对两个数，相同返回true不同返回false，tf.argmax(y，1)返回y最大时对应的x
accuracy = tf.reduce_mean(tf.cast(correct_prediction,tf.float32))
#tf.cast()将上述结果每一次转换成浮点型，累加并取平均值，得到准确率

with tf.Session() as sess:
    sess.run(init)
    for epoch in range (21):
        for batch in range(n_batch):
            batch_xs,batch_ys = mnist.train.next_batch(batch_size)#
            sess.run(train_step, feed_dict={x:batch_xs,y:batch_ys,keep_prob:0.7})
        
        acc1 = sess.run(accuracy, feed_dict={x:mnist.test.images, y:mnist.test.labels,keep_prob:0.7})#每训练一次输出一次准确率，利用的是测试集的数据
        acc2 = sess.run(accuracy,feed_dict={x:mnist.train.images,y:mnist.train.labels,keep_prob:0.7})#显示利用训练集的训练精度
        print("Iter"+str(epoch)+",Testing Accuracy"+str(acc1)+",Training accuracy"+str(acc2))



结果：
Iter0,Testing Accuracy0.9127,Training accuracy0.912964
Iter1,Testing Accuracy0.9205,Training accuracy0.924709
Iter2,Testing Accuracy0.9288,Training accuracy0.928018
Iter3,Testing Accuracy0.9343,Training accuracy0.937164
Iter4,Testing Accuracy0.9292,Training accuracy0.932309
Iter5,Testing Accuracy0.9425,Training accuracy0.944818
Iter6,Testing Accuracy0.9422,Training accuracy0.948145
Iter7,Testing Accuracy0.9455,Training accuracy0.947855
Iter8,Testing Accuracy0.9477,Training accuracy0.951473
Iter9,Testing Accuracy0.9493,Training accuracy0.953745
Iter10,Testing Accuracy0.9476,Training accuracy0.952473
Iter11,Testing Accuracy0.9488,Training accuracy0.952527
Iter12,Testing Accuracy0.9474,Training accuracy0.954145
Iter13,Testing Accuracy0.9525,Training accuracy0.9554
Iter14,Testing Accuracy0.9491,Training accuracy0.954673
Iter15,Testing Accuracy0.953,Training accuracy0.959745
Iter16,Testing Accuracy0.9533,Training accuracy0.956164
Iter17,Testing Accuracy0.956,Training accuracy0.959291
Iter18,Testing Accuracy0.9541,Training accuracy0.9568
Iter19,Testing Accuracy0.9551,Training accuracy0.959745
Iter20,Testing Accuracy0.957,Training accuracy0.9606

问题：

• Jupyter中，利用shift+tab可以查看函数的使用方法，光标要放在括号间；
• InternalError: Blas GEMM launch failed ，不知道什么原因，但是重启了Jupyter就可以正常运行