上文提到,到目前为止,caffe总共提供了六种优化方法:
- Stochastic Gradient Descent (
type: "SGD"
), - AdaDelta (
type: "AdaDelta"
), - Adaptive Gradient (
type: "AdaGrad"
), - Adam (
type: "Adam"
), - Nesterov’s Accelerated Gradient (
type: "Nesterov"
) and - RMSprop (
type: "RMSProp"
)
Solver就是用来使loss最小化的优化方法。对于一个数据集D,需要优化的目标函数是整个数据集中所有数据loss的平均值。
其中,fW(x(i))计算的是数据x(i)上的loss, 先将每个单独的样本x的loss求出来,然后求和,最后求均值。 r(W)是正则项(weight_decay),为了减弱过拟合现象。
如果采用这种Loss 函数,迭代一次需要计算整个数据集,在数据集非常大的这情况下,这种方法的效率很低,这个也是我们熟知的梯度下降采用的方法。
在实际中,通过将整个数据集分成几批(batches), 每一批就是一个mini-batch,其数量(batch_size)为N<<|D|,此时的loss 函数为:
有了loss函数后,就可以迭代的求解loss和梯度来优化这个问题。在神经网络中,用forward pass来求解loss,用backward pass来求解梯度。
在caffe中,默认采用的Stochastic Gradient Descent(SGD)进行优化求解。后面几种方法也是基于梯度的优化方法(like SGD),因此本文只介绍一下SGD。其它的方法,有兴趣的同学,可以去看文献原文。
1、Stochastic gradient descent(SGD)
随机梯度下降(Stochastic gradient descent)是在梯度下降法(gradient descent)的基础上发展起来的,梯度下降法也叫最速下降法,具体原理在网易公开课《机器学习》中,吴恩达教授已经讲解得非常详细。SGD在通过负梯度和上一次的权重更新值Vt的线性组合来更新W,迭代公式如下:
其中, 是负梯度的学习率(base_lr),
是上一次梯度值的权重(momentum),用来加权之前梯度方向对现在梯度下降方向的影响。这两个参数需要通过tuning来得到最好的结果,一般是根据经验设定的。如果你不知道如何设定这些参数,可以参考相关的论文。
在深度学习中使用SGD,比较好的初始化参数的策略是把学习率设为0.01左右(base_lr: 0.01),在训练的过程中,如果loss开始出现稳定水平时,对学习率乘以一个常数因子(gamma),这样的过程重复多次。
对于momentum,一般取值在0.5–0.99之间。通常设为0.9,momentum可以让使用SGD的深度学习方法更加稳定以及快速。
关于更多的momentum,请参看Hinton的《A Practical Guide to Training Restricted Boltzmann Machines》。
实例:
base_lr: 0.01 lr_policy: "step" gamma: 0.1 stepsize: 1000 max_iter: 3500 momentum: 0.9
lr_policy设置为step,则学习率的变化规则为 base_lr * gamma ^ (floor(iter / stepsize))
即前1000次迭代,学习率为0.01; 第1001-2000次迭代,学习率为0.001; 第2001-3000次迭代,学习率为0.00001,第3001-3500次迭代,学习率为10-5
上面的设置只能作为一种指导,它们不能保证在任何情况下都能得到最佳的结果,有时候这种方法甚至不work。如果学习的时候出现diverge(比如,你一开始就发现非常大或者NaN或者inf的loss值或者输出),此时你需要降低base_lr的值(比如,0.001),然后重新训练,这样的过程重复几次直到你找到可以work的base_lr。
2、AdaDelta
AdaDelta是一种”鲁棒的学习率方法“,是基于梯度的优化方法(like SGD)。
具体的介绍文献:
M. Zeiler ADADELTA: AN ADAPTIVE LEARNING RATE METHOD. arXiv preprint, 2012.
示例:
net: "examples/mnist/lenet_train_test.prototxt" test_iter: 100 test_interval: 500 base_lr: 1.0 lr_policy: "fixed" momentum: 0.95 weight_decay: 0.0005 display: 100 max_iter: 10000 snapshot: 5000 snapshot_prefix: "examples/mnist/lenet_adadelta" solver_mode: GPU type: "AdaDelta" delta: 1e-6
从最后两行可看出,设置solver type为Adadelta时,需要设置delta的值。
3、AdaGrad
自适应梯度(adaptive gradient)是基于梯度的优化方法(like SGD)
具体的介绍文献:
Duchi, E. Hazan, and Y. Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. The Journal of Machine Learning Research, 2011.
示例:
net: "examples/mnist/mnist_autoencoder.prototxt" test_state: { stage: 'test-on-train' } test_iter: 500 test_state: { stage: 'test-on-test' } test_iter: 100 test_interval: 500 test_compute_loss: true base_lr: 0.01 lr_policy: "fixed" display: 100 max_iter: 65000 weight_decay: 0.0005 snapshot: 10000 snapshot_prefix: "examples/mnist/mnist_autoencoder_adagrad_train" # solver mode: CPU or GPU solver_mode: GPU type: "AdaGrad"
4、Adam
是一种基于梯度的优化方法(like SGD)。
具体的介绍文献:
D. Kingma, J. Ba. Adam: A Method for Stochastic Optimization. International Conference for Learning Representations, 2015.
5、NAG
Nesterov 的加速梯度法(Nesterov’s accelerated gradient)作为凸优化中最理想的方法,其收敛速度非常快。
具体的介绍文献:
I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the Importance of Initialization and Momentum in Deep Learning. Proceedings of the 30th International Conference on Machine Learning, 2013.
示例:
net: "examples/mnist/mnist_autoencoder.prototxt" test_state: { stage: 'test-on-train' } test_iter: 500 test_state: { stage: 'test-on-test' } test_iter: 100 test_interval: 500 test_compute_loss: true base_lr: 0.01 lr_policy: "step" gamma: 0.1 stepsize: 10000 display: 100 max_iter: 65000 weight_decay: 0.0005 snapshot: 10000 snapshot_prefix: "examples/mnist/mnist_autoencoder_nesterov_train" momentum: 0.95 # solver mode: CPU or GPU solver_mode: GPU type: "Nesterov"
6、RMSprop
RMSprop是Tieleman在一次 Coursera课程演讲中提出来的,也是一种基于梯度的优化方法(like SGD)
具体的介绍文献:
T. Tieleman, and G. Hinton. RMSProp: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning.Technical report, 2012.
示例:
net: "examples/mnist/lenet_train_test.prototxt" test_iter: 100 test_interval: 500 base_lr: 1.0 lr_policy: "fixed" momentum: 0.95 weight_decay: 0.0005 display: 100 max_iter: 10000 snapshot: 5000 snapshot_prefix: "examples/mnist/lenet_adadelta" solver_mode: GPU type: "RMSProp" rms_decay: 0.98
最后两行,需要设置rms_decay值。
SGD
- x+= -learning_rate*dx
x+= -learning_rate*dx
Momentum
Momentum可以使SGD不至于陷入局部鞍点震荡,同时起到一定加速作用。
Momentum最开始有可能会偏离较远(overshooting the target),但是通常会慢慢矫正回来。
- v = mu*v - learning_rate*dx
- x+= v
v = mu*v - learning_rate*dx
x+= v
Nesterov momentum
基本思路是每次不在x位置求dx,而是在x+mu*v处更新dx,然后在用动量公式进行计算
相当于每次先到动量的位置,然后求梯度更新
vt=μvt?1?ε▽f(θt?1+μvt?1)
θt=θt?1+vt
计算▽f(θt?1+μvt?1)不太方便,做如下变量替换:?t?1=θt?1+μvt?1 ,并带回上述公式可以得到
vt=μvt?1+ε▽f(?t?1)
?t?1=?t?1?μvt?1+(1+μ)vt
- v_prev = v
- v = mu*v-learning_rate*dx
- x += -mu*v_prev+(1+mu)*v
v_prev = v
v = mu*v-learning_rate*dx
x += -mu*v_prev+(1+mu)*v
AdaGrad
使用每个变量的历史梯度值累加作为更新的分母,起到平衡不同变量梯度数值差异过大的问题
- cache += dx**2
- x += -learning_rate*dx/(np.sqrt(cache)+1e-7)
cache += dx**2
x += -learning_rate*dx/(np.sqrt(cache)+1e-7)
RMSProp
在AdaGrad基础上加入了decay factor,防止历史梯度求和过大
- cache = decay_rate*cache + (1-decay_rate)*dx**2
- x += -learning_rate*dx/(np.sqrt(cache)+1e-7)
cache = decay_rate*cache + (1-decay_rate)*dx**2
x += -learning_rate*dx/(np.sqrt(cache)+1e-7)
ADAM
初始版本:类似于加入动量的RMSProp
- m = beta1*m + (1-beta1)*dx
- v = beta2*v + (1-beta2)*(dx**2)
- x += -learning_rate*m / (np.sqrt(v)+1e-7)
m = beta1*m + (1-beta1)*dx
v = beta2*v + (1-beta2)*(dx**2)
x += -learning_rate*m / (np.sqrt(v)+1e-7)
真实的更新算法如下:
- m = beta1*m + (1-beta1)*dx
- v = beta2*v + (1-beta2)*(dx**2)
- mb = m/(1-beta1**t) # t is step number
- vb = v/(1-beta2**t)
- x += -learning_rate*mb / (np.sqrt(vb)+1e-7)
m = beta1*m + (1-beta1)*dx
v = beta2*v + (1-beta2)*(dx**2)
mb = m/(1-beta1**t) # t is step number
vb = v/(1-beta2**t)
x += -learning_rate*mb / (np.sqrt(vb)+1e-7)
mb和vb起到最开始的时候warm up作用,t很大之后(1-beta1**t) =1
Second Order optimization methods
second-order taylor expansion:
J(θ)≈J(θ0)+(θ?theta0)T+12(θ?θ0)TH(θ?θ0)
θ?=θ0?H?1▽θJ(θ0)
Does not form/store the full inverse Hessian.
Usually works very well in full batch, deterministic mode
实际经验
ADAM通常会取得比较好的结果,同时收敛非常快相比SGD L-BFGS适用于全batch做优化的情况 有时候可以多种优化方法同时使用,比如使用SGD进行warm up,然后ADAM 对于比较奇怪的需求,deepbit两个loss的收敛需要进行控制的情况,比较慢的SGD比较适用tensorflow 不同优化算法对应的参数
SGD
optimizer = tf.train.GradientDescentOptimizer(learning_rate=self.learning_rate)
Momentum
optimizer = tf.train.MomentumOptimizer(lr, 0.9)
AdaGrad
optimizer = tf.train.AdagradientOptimizer(learning_rate=self.learning_rate)
RMSProp
optimizer = tf.train.RMSPropOptimizer(0.001, 0.9)
ADAM
optimizer = tf.train.AdamOptimizer(learning_rate=self.learning_rate, epsilon=1e-08)
部分局部参数需要查找tensorflow官方文档
直接进行优化
train_op = optimizer.minimize(loss)
获得提取进行截断等处理
gradients, v = zip(*optimizer.compute_gradients(loss))
gradients, _ = tf.clip_by_global_norm(gradients, self.max_gradient_norm)
train_op = optimizer.apply_gradients(zip(gradients, v), global_step=self.global_step)
Caffe 不同优化算法参数
caffe的优化需要在solver.prototxt中指定相应的参数
type代表的是优化算法
比较坑的是不同的版本之间type会有变化(ADAM or Adam),需要看具体代码
* Stochastic Gradient Descent (type: “SGD”),
* AdaDelta (type: “AdaDelta”),
* Adaptive Gradient (type: “AdaGrad”),
* Adam (type: “Adam”),
* Nesterov’s Accelerated Gradient (type: “Nesterov”) and
* RMSprop (type: “RMSProp”)
SGD
- base_lr: 0.01
- lr_policy: ”step” # 也可以使用指数,多项式等等
- gamma: 0.1
- stepsize: 1000
- max_iter: 3500
- momentum: 0.9
base_lr: 0.01
lr_policy: "step" # 也可以使用指数,多项式等等
gamma: 0.1
stepsize: 1000
max_iter: 3500
momentum: 0.9
AdaDelta
- net: “examples/mnist/lenet_train_test.prototxt”
- test_iter: 100
- test_interval: 500
- base_lr: 1.0
- lr_policy: ”fixed”
- momentum: 0.95
- weight_decay: 0.0005
- display: 100
- max_iter: 10000
- snapshot: 5000
- snapshot_prefix: ”examples/mnist/lenet_adadelta”
- solver_mode: GPU
- type: ”AdaDelta”
- delta: 1e-6
net: "examples/mnist/lenet_train_test.prototxt"
test_iter: 100
test_interval: 500
base_lr: 1.0
lr_policy: "fixed"
momentum: 0.95
weight_decay: 0.0005
display: 100
max_iter: 10000
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet_adadelta"
solver_mode: GPU
type: "AdaDelta"
delta: 1e-6
AdaGrad
- net: “examples/mnist/mnist_autoencoder.prototxt”
- test_state: { stage: 'test-on-train' }
- test_iter: 500
- test_state: { stage: 'test-on-test' }
- test_iter: 100
- test_interval: 500
- test_compute_loss: true
- base_lr: 0.01
- lr_policy: ”fixed”
- display: 100
- max_iter: 65000
- weight_decay: 0.0005
- snapshot: 10000
- snapshot_prefix: ”examples/mnist/mnist_autoencoder_adagrad_train”
- # solver mode: CPU or GPU
- solver_mode: GPU
- type: ”AdaGrad”
net: "examples/mnist/mnist_autoencoder.prototxt"
test_state: { stage: 'test-on-train' }
test_iter: 500
test_state: { stage: 'test-on-test' }
test_iter: 100
test_interval: 500
test_compute_loss: true
base_lr: 0.01
lr_policy: "fixed"
display: 100
max_iter: 65000
weight_decay: 0.0005
snapshot: 10000
snapshot_prefix: "examples/mnist/mnist_autoencoder_adagrad_train"
solver mode: CPU or GPU
solver_mode: GPU
type: "AdaGrad"
Nesterov
- base_lr: 0.01
- lr_policy: ”step”
- gamma: 0.1
- weight_decay: 0.0005
- momentum: 0.95
- type: ”Nesterov”
base_lr: 0.01
lr_policy: "step"
gamma: 0.1
weight_decay: 0.0005
momentum: 0.95
type: "Nesterov"
ADAM
- train_net: “nin_train_val.prototxt”
- base_lr: 0.001
- ###############
- ##### step:base_lr * gamma ^ (floor(iter / stepsize))
- #lr_policy: ”step”
- #gamma: 0.1
- #stepsize: 25000
- ##### multi-step:
- #lr_policy: ”multistep”
- #gamma: 0.5
- #stepvalue: 1000
- #stepvalue: 2000
- #stepvalue: 3000
- #stepvalue: 4000
- #stepvalue: 5000
- #stepvalue: 10000
- #stepvalue: 20000
- ###### inv:base_lr * (1 + gamma * iter) ^ (- power)
- # lr_policy: ”inv”
- # gamma: 0.0001
- # power: 2
- ##### exp:base_lr * gamma ^ iter
- # lr_policy: ”exp”
- # gamma: 0.9
- ##### poly:base_lr (1 - iter/max_iter) ^ (power)
- # lr_policy: ”poly”
- # power: 0.9
- ##### sigmoid:base_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))
- # lr_policy: ”sigmoid”
- # gamma: 0.9
- #momentum: 0.9
- solver_type: ADAM
- momentum: 0.9
- momentum2: 0.999
- delta: 1e-8
- lr_policy: ”fixed”
- display: 100
- max_iter: 50000
- weight_decay: 0.0005
- snapshot: 5000
- snapshot_prefix: ”./stage1/sgd_DeepBit1024_alex_stage1”
- solver_mode: GPU
train_net: "nin_train_val.prototxt"
base_lr: 0.001
#
step:base_lr * gamma ^ (floor(iter / stepsize))
lr_policy: "step"
gamma: 0.1
stepsize: 25000
multi-step:
lr_policy: "multistep"
gamma: 0.5
stepvalue: 1000
stepvalue: 2000
stepvalue: 3000
stepvalue: 4000
stepvalue: 5000
stepvalue: 10000
stepvalue: 20000
inv:base_lr * (1 + gamma * iter) ^ (- power)
lr_policy: "inv"
gamma: 0.0001
power: 2
exp:base_lr * gamma ^ iter
lr_policy: "exp"
gamma: 0.9
poly:base_lr (1 - iter/max_iter) ^ (power)
lr_policy: "poly"
power: 0.9
sigmoid:base_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))
lr_policy: "sigmoid"
gamma: 0.9
momentum: 0.9
solver_type: ADAM
momentum: 0.9
momentum2: 0.999
delta: 1e-8
lr_policy: “fixed”
display: 100
max_iter: 50000
weight_decay: 0.0005
snapshot: 5000
snapshot_prefix: “./stage1/sgd_DeepBit1024_alex_stage1”
solver_mode: GPU
RMSProp
- net: “examples/mnist/lenet_train_test.prototxt”
- test_iter: 100
- test_interval: 500
- base_lr: 1.0
- lr_policy: ”fixed”
- momentum: 0.95
- weight_decay: 0.0005
- display: 100
- max_iter: 10000
- snapshot: 5000
- snapshot_prefix: ”examples/mnist/lenet_adadelta”
- solver_mode: GPU
- type: ”RMSProp”
- rms_decay: 0.98
net: "examples/mnist/lenet_train_test.prototxt"
test_iter: 100
test_interval: 500
base_lr: 1.0
lr_policy: "fixed"
momentum: 0.95
weight_decay: 0.0005
display: 100
max_iter: 10000
snapshot: 5000
snapshot_prefix: "examples/mnist/lenet_adadelta"
solver_mode: GPU
type: "RMSProp"
rms_decay: 0.98