优化之——学习率的设置

先用较大的学习率来加速得到较优解，随着迭代的继续逐步减小学习率，使得模型在训练后期更加稳定。
decayed_learning_rate = learning_rate*decay_rate^(global_step/decay_step)
learning_rate为设定的初始学习率
decay_rate为衰减系数
global_step已训练次数
decay_step为使用一遍训练数据需要迭代的论数
decay_step= train_num/batch_size

exponential_decay(learning_rate, global_step, decay_steps, decay_rate,
                      staircase=False, name=None):

global_step = tf.Variable(0)
learning_rate = tf.train.exponential_decay(LEARNING_RATE_BASE,
global_step,
mnist.train.num_examples/BATCH_SIZE,
LEARNING_RATE_DECAY)
train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss,global_step=global_step)
自动更新global_step, global_step: Optional `Variable` to increment by one after the
        variables have been updateapply_updates = state_ops.assign_add(global_step, 1, name=name).op

        for i in range(TRAINING_STEP):
            xs , ys = mnist.train.next_batch(BATCH_SIZE)
            sess.run(train_step,feed_dict={x:xs,y_:ys})