# 一文看懂常用的梯度下降算法

01

Momentum optimization

02

NAG

1 NAG效果图

03

04

RMSprop

05

3 不同学习速率的训练效果

TensorFlow中，你可以这样实现：

initial_learning_rate = 0.1decay_steps = 10000decay_rate = 1/10global_step = tf.Variable(0, trainable=False)learning_rate = tf.train.exponential_decay(initial_learning_rate,                                                       global_step, decay_steps, decay_rate)# decayed_learning_rate = learning_rate *#                decay_rate ^ (global_step / decay_steps)optimizer = tf.train.MomentumOptimizer(learning_rate, momentum=0.9)training_op = optimizer.minimize(loss, global_step=global_step)

2. Hands-OnMachine Learning with Scikit-Learn and TensorFlow, Aurélien Géron, 2017.

3. NAG:http://proceedings.mlr.press/v28/sutskever13.pdf.

5. RMSprop:http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf.

8. 不同的算法的效果可视化：https://imgur.com/a/Hqolp.

80%的AI从业者已关注我们微信公众号

11-03 3万+
01-04 2万+
01-21 14万+
11-29 5913
04-14 3万+
04-04 2272
06-05 3642
09-09 8119
08-03 6014
11-14 9614
07-05 7万+
09-07 6338
04-19 1万+
10-15 7303
10-07 235
07-31 2万+
09-09 6万+