剖析与总结
这次实验使用了四种下降的方式：GD、mini-batch GD、Momentum，Adam
稍微总结一下就是，GD可以保证每次都下降代价函数，但是每次迭代需要走完所有训练集，速度较慢；而随机梯度下降，相当于每次饲喂训练集进行优化，而这并不能保证对于所有训练集都在优化，所以比较曲曲折折，大差不差；mini-batch介于二者之间。
那么，怎么样又要获得mini-batch的高速度，又要让梯度下降变快呢，这时候就有了momentum、RMSprop，还有集二者于一身的Adam算法。
事实上，这三个算法的思路都很显然，就是在仿物理学，保持一个惯性在里面，当然这种取平均的方法也可以理解有灰度预测的意思。总之就是要使用原来保留有的v把曲曲折折的下降掰直过来。加速下降。
Gradient Descent
梯度下降代码
# GRADED FUNCTION: update_parameters_with_gd
def update_parameters_with_gd(parameters, grads, learning_rate):
"""
Update parameters using one step of gradient descent
Arguments:
parameters -- python dictionary containing your parameters to be updated:
parameters['W' + str(l)] = Wl
parameters['b' + str(l)] = bl
grads -- python dictionary containing your gradients to update each parameters:
grads['dW' + str(l)] = dWl
grads['db' + str(l)] = dbl
learning_rate -- the learning rate, scalar.
Returns:
parameters -- python dictionary containing your updated parameters
"""
L = len(parameters) // 2 # number of layers in the neural networks
# Update rule for each parameter
for l in range(L):
### START CODE HERE ### (approx. 2 lines)
parameters['W' + str(l+1)] -= learning_rate * grads['dW' + str(l+1)]
parameters['b' + str(l+1)] -= learning_rate * grads['db' + str(l+1)]
### END CODE HERE ###
return parameters
In Stochastic Gradient Descent, you use only 1 training example before updating the gradients. When the training set is large, SGD can be faster. But the parameters will “oscillate” toward the minimum rather than converge smoothly. Here is an illustration of this:
（在随机梯度下降中，在更新梯度之前，只使用1个训练样例。当训练集大时，SGD可以更快。但是这些参数会向最小值“摆动”而不是平稳地收敛。这是一个例子演示）
Figure 1 : SGD vs GD
“+” denotes a minimum of the cost. SGD leads to many oscillations to reach convergence. But each step is a lot faster to compute for SGD than for GD, as it uses only one training example (vs. the whole batch for GD).
Note also that implementing SGD requires 3 for-loops in total:
- Over the number of iterations
- Over the m m m training examples
- Over the layers (to update all parameters, from ( W [ 1 ] , b [ 1 ] ) (W^{[1]},b^{[1]}) (W[1],b[1]) to ( W [ L ] , b [ L ] ) (W^{[L]},b^{[L]}) (W[L],b[L]))
In practice, you’ll often get faster results if you do not use neither the whole training set, nor only one training example, to perform each update. Mini-batch gradient descent uses an intermediate number of examples for each step. With mini-batch gradient descent, you loop over the mini-batches instead of looping over individual training examples.
Figure 2: SGD vs Mini-Batch GD
“+” denotes a minimum of the cost. Using mini-batches in your optimization algorithm often leads to faster optimization.
What you should remember:
- The difference between gradient descent, mini-batch gradient descent and stochastic gradient descent is the number of examples you use to perform one update step.
- You have to tune a learning rate hyperparameter α \alpha α.
- With a well-turned mini-batch size, usually it outperforms either gradient descent or stochastic gradient descent (particularly when the training set is large).
Mini-Batch Gradient descent
Let’s learn how to build mini-batches from the training set (X, Y).
There are two steps:
- Shuffle: Create a shuffled version of the training set (X, Y) as shown below. Each column of X and Y represents a training example. Note that the random shuffling is done synchronously between X and Y. Such that after the shuffling the i t h i^{th} ith column of X is the example corresponding to the i t h i^{th} ith label in Y. The shuffling step ensures that examples will be split randomly into different mini-batches.
- Partition: Partition the shuffled (X, Y) into mini-batches of size
mini_batch_size(here 64). Note that the number of training examples is not always divisible bymini_batch_size. The last mini batch might be smaller, but you don’t need to worry about this. When the final mini-batch is smaller than the fullmini_batch_size, it will look like this:
Exercise: Implement random_mini_batches. We coded the shuffling part for you. To help you with the partitioning step, we give you the following code that selects the indexes for the
1
s
t
1^{st}
1st and
2
n
d
2^{nd}
2nd mini-batches:
first_mini_batch_X = shuffled_X[:, 0 : mini_batch_size]
second_mini_batch_X = shuffled_X[:, mini_batch_size : 2 * mini_batch_size]
...
Note that the last mini-batch might end up smaller than mini_batch_size=64. Let
⌊
s
⌋
\lfloor s \rfloor
⌊s⌋ represents
s
s
s rounded down to the nearest integer (this is math.floor(s) in Python). If the total number of examples is not a multiple of mini_batch_size=64 then there will be
⌊
m
m
i
n
i
_
b
a
t
c
h
_
s
i
z
e
⌋
\lfloor \frac{m}{mini\_batch\_size}\rfloor
⌊mini_batch_sizem⌋ mini-batches with a full 64 examples, and the number of examples in the final mini-batch will be (
m
−
m
i
n
i
_
b
a
t
c
h
_
s
i
z
e
×
⌊
m
m
i
n
i
_
b
a
t
c
h
_
s
i
z
e
⌋
m-mini_\_batch_\_size \times \lfloor \frac{m}{mini\_batch\_size}\rfloor
m−mini_batch_size×⌊mini_batch_sizem⌋).
小批次下降代码
# GRADED FUNCTION: random_mini_batches
def random_mini_batches(X, Y, mini_batch_size = 64, seed = 0):
"""
Creates a list of random minibatches from (X, Y)
Arguments:
X -- input data, of shape (input size, number of examples)
Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)
mini_batch_size -- size of the mini-batches, integer
Returns:
mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)
"""
np.random.seed(seed) # To make your "random" minibatches the same as ours
m = X.shape[1] # number of training examples
mini_batches = []
# Step 1: Shuffle (X, Y)
permutation = list(np.random.permutation(m))
shuffled_X = X[:, permutation]
shuffled_Y = Y[:, permutation].reshape((1,m))
# Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.
num_complete_minibatches = math.floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning
for k in range(0, num_complete_minibatches):
### START CODE HERE ### (approx. 2 lines)
mini_batch_X = shuffled_X[:,k*mini_batch_size:(k+1)*mini_batch_size]
mini_batch_Y = shuffled_Y[:,k*mini_batch_size:(k+1)*mini_batch_size]
### END CODE HERE ###
mini_batch = (mini_batch_X, mini_batch_Y)
mini_batches.append(mini_batch)
# Handling the end case (last mini-batch < mini_batch_size)
if m % mini_batch_size != 0:
### START CODE HERE ### (approx. 2 lines)
mini_batch_X = shuffled_X[:,num_complete_minibatches*mini_batch_size:]
mini_batch_Y = shuffled_Y[:,num_complete_minibatches*mini_batch_size:]
### END CODE HERE ###
mini_batch = (mini_batch_X, mini_batch_Y)
mini_batches.append(mini_batch)
return mini_batches
Momentum
Because mini-batch gradient descent makes a parameter update after seeing just a subset of examples, the direction of the update has some variance, and so the path taken by mini-batch gradient descent will “oscillate” toward convergence. Using momentum can reduce these oscillations.
Momentum takes into account the past gradients to smooth out the update. We will store the ‘direction’ of the previous gradients in the variable v v v. Formally, this will be the exponentially weighted average of the gradient on previous steps. You can also think of v v v as the “velocity” of a ball rolling downhill, building up speed (and momentum) according to the direction of the gradient/slope of the hill.
Figure 3: The red arrows shows the direction taken by one step of mini-batch gradient descent with momentum. The blue points show the direction of the gradient (with respect to the current mini-batch) on each step. Rather than just following the gradient, we let the gradient influence v v v and then take a step in the direction of v v v.
Exercise: Initialize the velocity. The velocity,
v
v
v, is a python dictionary that needs to be initialized with arrays of zeros. Its keys are the same as those in the grads dictionary, that is:
for
l
=
1
,
.
.
.
,
L
l =1,...,L
l=1,...,L:
v["dW" + str(l+1)] = ... #(numpy array of zeros with the same shape as parameters["W" + str(l+1)])
v["db" + str(l+1)] = ... #(numpy array of zeros with the same shape as parameters["b" + str(l+1)])
Note that the iterator l starts at 0 in the for loop while the first parameters are v[“dW1”] and v[“db1”] (that’s a “one” on the superscript). This is why we are shifting l to l+1 in the for loop.
木馒头代码
# GRADED FUNCTION: initialize_velocity
def initialize_velocity(parameters):
"""
Initializes the velocity as a python dictionary with:
- keys: "dW1", "db1", ..., "dWL", "dbL"
- values: numpy arrays of zeros of the same shape as the corresponding gradients/parameters.
Arguments:
parameters -- python dictionary containing your parameters.
parameters['W' + str(l)] = Wl
parameters['b' + str(l)] = bl
Returns:
v -- python dictionary containing the current velocity.
v['dW' + str(l)] = velocity of dWl
v['db' + str(l)] = velocity of dbl
"""
L = len(parameters) // 2 # number of layers in the neural networks
v = {}
# Initialize velocity
for l in range(L):
### START CODE HERE ### (approx. 2 lines)
v["dW" + str(l+1)] = np.zeros((parameters['W'+str(l+1)].shape[0],parameters['W'+str(l+1)].shape[1])) #(numpy array of zeros with the same shape as parameters["W" + str(l+1)])
v["db" + str(l+1)] = np.zeros((parameters['b'+str(l+1)].shape[0],parameters['b'+str(l+1)].shape[1])) #(numpy array of zeros with the same shape as parameters["b" + str(l+1)])
### END CODE HERE ###
return v
# GRADED FUNCTION: update_parameters_with_momentum
def update_parameters_with_momentum(parameters, grads, v, beta, learning_rate):
"""
Update parameters using Momentum
Arguments:
parameters -- python dictionary containing your parameters:
parameters['W' + str(l)] = Wl
parameters['b' + str(l)] = bl
grads -- python dictionary containing your gradients for each parameters:
grads['dW' + str(l)] = dWl
grads['db' + str(l)] = dbl
v -- python dictionary containing the current velocity:
v['dW' + str(l)] = ...
v['db' + str(l)] = ...
beta -- the momentum hyperparameter, scalar
learning_rate -- the learning rate, scalar
Returns:
parameters -- python dictionary containing your updated parameters
v -- python dictionary containing your updated velocities
"""
L = len(parameters) // 2 # number of layers in the neural networks
# Momentum update for each parameter
for l in range(L):
### START CODE HERE ### (approx. 4 lines)
# compute velocities
v['dW' + str(l+1)] = beta*v['dW' + str(l+1)] + (1-beta)*grads['dW' + str(l+1)]
v['db' + str(l+1)] = beta*v['db' + str(l+1)] + (1-beta)*grads['db' + str(l+1)]
# update parameters
parameters['W'+str(l+1)] -= learning_rate * v['dW' + str(l+1)]
parameters['b'+str(l+1)] -= learning_rate * v['db' + str(l+1)]
### END CODE HERE ###
return parameters, v
Adam
Adam is one of the most effective optimization algorithms for training neural networks. It combines ideas from RMSProp (described in lecture) and Momentum.
How does Adam work?
- It calculates an exponentially weighted average of past gradients, and stores it in variables v v v (before bias correction) and v c o r r e c t e d v^{corrected} vcorrected (with bias correction).
- It calculates an exponentially weighted average of the squares of the past gradients, and stores it in variables s s s (before bias correction) and s c o r r e c t e d s^{corrected} scorrected (with bias correction).
- It updates parameters in a direction based on combining information from “1” and “2”.
The update rule is, for l = 1 , . . . , L l = 1, ..., L l=1,...,L:
{
v
d
W
[
l
]
=
β
1
v
d
W
[
l
]
+
(
1
−
β
1
)
∂
J
∂
W
[
l
]
v
d
W
[
l
]
c
o
r
r
e
c
t
e
d
=
v
d
W
[
l
]
1
−
(
β
1
)
t
s
d
W
[
l
]
=
β
2
s
d
W
[
l
]
+
(
1
−
β
2
)
(
∂
J
∂
W
[
l
]
)
2
s
d
W
[
l
]
c
o
r
r
e
c
t
e
d
=
s
d
W
[
l
]
1
−
(
β
1
)
t
W
[
l
]
=
W
[
l
]
−
α
v
d
W
[
l
]
c
o
r
r
e
c
t
e
d
s
d
W
[
l
]
c
o
r
r
e
c
t
e
d
+
ε
\begin{cases} v_{dW^{[l]}} = \beta_1 v_{dW^{[l]}} + (1 - \beta_1) \frac{\partial \mathcal{J} }{ \partial W^{[l]} } \\ v^{corrected}_{dW^{[l]}} = \frac{v_{dW^{[l]}}}{1 - (\beta_1)^t} \\ s_{dW^{[l]}} = \beta_2 s_{dW^{[l]}} + (1 - \beta_2) (\frac{\partial \mathcal{J} }{\partial W^{[l]} })^2 \\ s^{corrected}_{dW^{[l]}} = \frac{s_{dW^{[l]}}}{1 - (\beta_1)^t} \\ W^{[l]} = W^{[l]} - \alpha \frac{v^{corrected}_{dW^{[l]}}}{\sqrt{s^{corrected}_{dW^{[l]}}} + \varepsilon} \end{cases}
⎩⎪⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎪⎧vdW[l]=β1vdW[l]+(1−β1)∂W[l]∂JvdW[l]corrected=1−(β1)tvdW[l]sdW[l]=β2sdW[l]+(1−β2)(∂W[l]∂J)2sdW[l]corrected=1−(β1)tsdW[l]W[l]=W[l]−αsdW[l]corrected+εvdW[l]corrected
where:
- t counts the number of steps taken of Adam
- L is the number of layers
- β 1 \beta_1 β1 and β 2 \beta_2 β2 are hyperparameters that control the two exponentially weighted averages.
- α \alpha α is the learning rate
- ε \varepsilon ε is a very small number to avoid dividing by zero
As usual, we will store all parameters in the parameters dictionary
亚当代码
# GRADED FUNCTION: initialize_adam
def initialize_adam(parameters) :
"""
Initializes v and s as two python dictionaries with:
- keys: "dW1", "db1", ..., "dWL", "dbL"
- values: numpy arrays of zeros of the same shape as the corresponding gradients/parameters.
Arguments:
parameters -- python dictionary containing your parameters.
parameters["W" + str(l)] = Wl
parameters["b" + str(l)] = bl
Returns:
v -- python dictionary that will contain the exponentially weighted average of the gradient.
v["dW" + str(l)] = ...
v["db" + str(l)] = ...
s -- python dictionary that will contain the exponentially weighted average of the squared gradient.
s["dW" + str(l)] = ...
s["db" + str(l)] = ...
"""
L = len(parameters) // 2 # number of layers in the neural networks
v = {}
s = {}
# Initialize v, s. Input: "parameters". Outputs: "v, s".
for l in range(L):
### START CODE HERE ### (approx. 4 lines)
v["dW" + str(l+1)] = np.zeros((parameters["W" + str(l+1)].shape[0],parameters["W" + str(l+1)].shape[1])) #(numpy array of zeros with the same shape as parameters["W" + str(l+1)])
v["db" + str(l+1)] = np.zeros((parameters["b" + str(l+1)].shape[0],parameters["b" + str(l+1)].shape[1])) #(numpy array of zeros with the same shape as parameters["b" + str(l+1)])
s["dW" + str(l+1)] = np.zeros((parameters["W" + str(l+1)].shape[0],parameters["W" + str(l+1)].shape[1])) #(numpy array of zeros with the same shape as parameters["W" + str(l+1)])
s["db" + str(l+1)] = np.zeros((parameters["b" + str(l+1)].shape[0],parameters["b" + str(l+1)].shape[1])) #(numpy array of zeros with the same shape as parameters["b" + str(l+1)])
### END CODE HERE ###
return v, s
# GRADED FUNCTION: update_parameters_with_adam
def update_parameters_with_adam(parameters, grads, v, s, t, learning_rate = 0.01,
beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8):
"""
Update parameters using Adam
Arguments:
parameters -- python dictionary containing your parameters:
parameters['W' + str(l)] = Wl
parameters['b' + str(l)] = bl
grads -- python dictionary containing your gradients for each parameters:
grads['dW' + str(l)] = dWl
grads['db' + str(l)] = dbl
v -- Adam variable, moving average of the first gradient, python dictionary
s -- Adam variable, moving average of the squared gradient, python dictionary
learning_rate -- the learning rate, scalar.
beta1 -- Exponential decay hyperparameter for the first moment estimates
beta2 -- Exponential decay hyperparameter for the second moment estimates
epsilon -- hyperparameter preventing division by zero in Adam updates
Returns:
parameters -- python dictionary containing your updated parameters
v -- Adam variable, moving average of the first gradient, python dictionary
s -- Adam variable, moving average of the squared gradient, python dictionary
"""
L = len(parameters) // 2 # number of layers in the neural networks
v_corrected = {} # Initializing first moment estimate, python dictionary
s_corrected = {} # Initializing second moment estimate, python dictionary
# Perform Adam update on all parameters
for l in range(L):
# Moving average of the gradients. Inputs: "v, grads, beta1". Output: "v".
### START CODE HERE ### (approx. 2 lines)
v['dW'+str(l+1)] = beta1*v['dW'+str(l+1)] + (1-beta1)*grads['dW'+str(l+1)]
v['db'+str(l+1)] = beta1*v['db'+str(l+1)] + (1-beta1)*grads['db'+str(l+1)]
### END CODE HERE ###
# Compute bias-corrected first moment estimate. Inputs: "v, beta1, t". Output: "v_corrected".
### START CODE HERE ### (approx. 2 lines)
v_corrected['dW'+str(l+1)] = v['dW'+str(l+1)]/(1 - beta1**t)
v_corrected['db'+str(l+1)] = v['db'+str(l+1)]/(1 - beta1**t)
### END CODE HERE ###
# Moving average of the squared gradients. Inputs: "s, grads, beta2". Output: "s".
### START CODE HERE ### (approx. 2 lines)
s['dW'+str(l+1)] = beta2*s['dW'+str(l+1)] + (1-beta2)*(np.square(grads['dW'+str(l+1)]))
s['db'+str(l+1)] = beta2*s['db'+str(l+1)] + (1-beta2)*(np.square(grads['db'+str(l+1)]))
### END CODE HERE ###
# Compute bias-corrected second raw moment estimate. Inputs: "s, beta2, t". Output: "s_corrected".
### START CODE HERE ### (approx. 2 lines)
s_corrected['dW'+str(l+1)] = s['dW'+str(l+1)]/(1 - beta2**t)
s_corrected['db'+str(l+1)] = s['db'+str(l+1)]/(1 - beta2**t)
### END CODE HERE ###
# Update parameters. Inputs: "parameters, learning_rate, v_corrected, s_corrected, epsilon". Output: "parameters".
### START CODE HERE ### (approx. 2 lines)
parameters['W'+str(l+1)] -= learning_rate*(v_corrected['dW'+str(l+1)])/(epsilon+np.sqrt(s_corrected['dW'+str(l+1)]) )
parameters['b'+str(l+1)] -= learning_rate*(v_corrected['db'+str(l+1)])/(epsilon+np.sqrt(s_corrected['db'+str(l+1)]) )
### END CODE HERE ###
return parameters, v, s
评测与结果
Mini-batch
Mini-batch with momentum
Mini-batch with Adam mode
Summary
| optimization method | accuracy | cost shape |
|---|---|---|
| Gradient descent | 79.70% | oscillations |
| Momentum | 79.70% | oscillations |
| Adam | 94% | smoother |
Momentum usually helps, but given the small learning rate and the simplistic dataset, its impact is almost negligeable. Also, the huge oscillations you see in the cost come from the fact that some minibatches are more difficult thans others for the optimization algorithm.
Adam on the other hand, clearly outperforms mini-batch gradient descent and Momentum. If you run the model for more epochs on this simple dataset, all three methods will lead to very good results. However, you’ve seen that Adam converges a lot faster.
Some advantages of Adam include:
- Relatively low memory requirements (though higher than gradient descent and gradient descent with momentum)
- Usually works well even with little tuning of hyperparameters (except α \alpha α)
275
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
