tensorflow learning_rate 的设置方法小结

1. 分段式learning rate的设置方式

其中

tf.app.flags.DEFINE_float('learning_rate', 1e-3, 'Initial learning rate.')
tf.app.flags.DEFINE_float('end_learning_rate', 0.000001,
                          'The minimal end learning rate used by a polynomial decay learning rate.')
# for learning rate piecewise_constant decay
tf.app.flags.DEFINE_string('decay_boundaries', '2000,80000, 100000',
                           'Learning rate decay boundaries by global_step (comma-separated list).')
tf.app.flags.DEFINE_string('lr_decay_factors', '0.1,1, 0.1, 0.01',
                           'The values of learning_rate decay factor for each segment between boundaries (comma-separated list).')

global_step = tf.train.get_or_create_global_step()

lr_values = [params['learning_rate'] * decay for decay in params['lr_decay_factors']]
learning_rate = tf.train.piecewise_constant(tf.cast(global_step, tf.int32),
                                            [int(_) for _ in params['decay_boundaries']],
                                            lr_values)
truncated_learning_rate = tf.maximum(learning_rate, tf.constant(params['end_learning_rate'], dtype=learning_rate.dtype),
                                     name='learning_rate')
# Create a tensor named learning_rate for logging purposes.
tf.summary.scalar('learning_rate', truncated_learning_rate)

注释：

Example: use a learning rate that's 1.0 for the first 100001 steps, 0.5 for the next 10000 steps, and 0.1 for any additional steps.

global_step = tf.Variable(0, trainable=Falseboundaries = [100000, 110000values = [1.0, 0.5, 0.1]learning_rate = tf.train.piecewise_constant(global_step, boundaries, value

2. 指数的方式