剖析与心得
在本次实验中,我们使用梯度检验方法,将输入的参数稍稍改变一点,因为我们有很大把握使得最后的损失值是对的,所以,我们直接用损失值的变化量除以自变量的改变量,得到的就是导数值的近似值,在一定误差允许范围内,如果其和我们使用梯度下降求出来的值相近,那么说明,我们梯度下降的算法的书写没有问题;否则则说明有问题。
本次实验中,写了两部分代码:
第一部分是针对简单的单元变量进行检验,热身;
第二部分是对于神经网络进行梯度检验。
注意理解抽象含义:对于某一个变量的偏导数就是所谓的梯度的一部分,而偏导数实则近似的话来看就是应变量(代价函数)变化量/自变量变化量。
(补充一点,在这个实验里面,最后按照他的地方去修改是改不到1e-7的…而是这个数,注意一下就好,没啥大事其实
ok现在开始
梯度检验原理(题干引入)
反向传播计算梯度 ∂ J ∂ θ \frac{\partial J}{\partial \theta} ∂θ∂J,其中 θ \theta θ表示模型的参数。使用正向传播和损失函数来计算 J J J。
由于正向传播相对容易实现,相信你有信心能做到这一点,确定100%计算正确的损失 J J J。为此,你可以使用 J J J来验证代码 ∂ J ∂ θ \frac{\partial J}{\partial \theta} ∂θ∂J。
让我们回顾一下导数(或者说梯度)的定义:
∂
J
∂
θ
=
lim
ε
→
0
J
(
θ
+
ε
)
−
J
(
θ
−
ε
)
2
ε
(1)
\frac{\partial J}{\partial \theta} = \lim_{\varepsilon \to 0} \frac{J(\theta + \varepsilon) - J(\theta - \varepsilon)}{2 \varepsilon} \tag{1}
∂θ∂J=ε→0lim2εJ(θ+ε)−J(θ−ε)(1)
如果你还不熟悉" lim ε → 0 \displaystyle \lim_{\varepsilon \to 0} ε→0lim"表示法,其意思只是“当 ε \varepsilon ε值趋向很小时”。
我们知道以下内容:
- ∂ J ∂ θ \frac{\partial J}{\partial \theta} ∂θ∂J是你要确保计算正确的对象。
- 你可以计算 J ( θ + ε ) J(\theta + \varepsilon) J(θ+ε)和 J ( θ − ε ) J(\theta - \varepsilon) J(θ−ε)(在 θ \theta θ是实数的情况下),因为要保证 J J J的实现是正确的。
让我们使用方程式(1)和 ε \varepsilon ε的一个小值来说服CEO你计算 ∂ J ∂ θ \frac{\partial J}{\partial \theta} ∂θ∂J的代码是正确的
一维部分的代码
# GRADED FUNCTION: forward_propagation
def forward_propagation(x, theta):
"""
Implement the linear forward propagation (compute J) presented in Figure 1 (J(theta) = theta * x)
Arguments:
x -- a real-valued input
theta -- our parameter, a real number as well
Returns:
J -- the value of function J, computed using the formula J(theta) = theta * x
"""
### START CODE HERE ### (approx. 1 line)
J = theta*x
### END CODE HERE ###
return J
# GRADED FUNCTION: backward_propagation
def backward_propagation(x, theta):
"""
Computes the derivative of J with respect to theta (see Figure 1).
Arguments:
x -- a real-valued input
theta -- our parameter, a real number as well
Returns:
dtheta -- the gradient of the cost with respect to theta
"""
### START CODE HERE ### (approx. 1 line)
dtheta = x
### END CODE HERE ###
return dtheta
练习:为了展示backward_propagation()
函数正确计算了梯度
∂
J
∂
θ
\frac{\partial J}{\partial \theta}
∂θ∂J,让我们实施梯度检验。
说明:
- 首先使用上式(1)和
ε
\varepsilon
ε的极小值计算“gradapprox”。以下是要遵循的步骤:
- θ + = θ + ε \theta^{+} = \theta + \varepsilon θ+=θ+ε
- θ − = θ − ε \theta^{-} = \theta - \varepsilon θ−=θ−ε
- J + = J ( θ + ) J^{+} = J(\theta^{+}) J+=J(θ+)
- J − = J ( θ − ) J^{-} = J(\theta^{-}) J−=J(θ−)
- g r a d a p p r o x = J + − J − 2 ε gradapprox = \frac{J^{+} - J^{-}}{2 \varepsilon} gradapprox=2εJ+−J−
- 然后使用反向传播计算梯度,并将结果存储在变量“grad”中
- 最后,使用以下公式计算“gradapprox”和“grad”之间的相对差:
d i f f e r e n c e = ∣ ∣ g r a d − g r a d a p p r o x ∣ ∣ 2 ∣ ∣ g r a d ∣ ∣ 2 + ∣ ∣ g r a d a p p r o x ∣ ∣ 2 (2) difference = \frac {\mid\mid grad - gradapprox \mid\mid_2}{\mid\mid grad \mid\mid_2 + \mid\mid gradapprox \mid\mid_2} \tag{2} difference=∣∣grad∣∣2+∣∣gradapprox∣∣2∣∣grad−gradapprox∣∣2(2)
你需要3个步骤来计算此公式:
* 1. 使用np.linalg.norm(…)计算分子
* 2. 计算分母,调用np.linalg.norm(…)两次
* 3. 相除
- 如果差异很小(例如小于 1 0 − 7 10^{-7} 10−7),则可以确信正确计算了梯度。否则,梯度计算可能会出错。
# GRADED FUNCTION: gradient_check
def gradient_check(x, theta, epsilon = 1e-7):
"""
Implement the backward propagation presented in Figure 1.
Arguments:
x -- a real-valued input
theta -- our parameter, a real number as well
epsilon -- tiny shift to the input to compute approximated gradient with formula(1)
Returns:
difference -- difference (2) between the approximated gradient and the backward propagation gradient
"""
# Compute gradapprox using left side of formula (1). epsilon is small enough, you don't need to worry about the limit.
### START CODE HERE ### (approx. 5 lines)
thetaplus = theta + epsilon # Step 1
thetaminus = theta - epsilon # Step 2
Jplus = forward_propagation(x,thetaplus) # Step 3
Jminus = forward_propagation(x,thetaminus) # Step 4
gradapprox = (Jplus - Jminus)/(2*epsilon) # Step 5
### END CODE HERE ###
# Check if gradapprox is close enough to the output of backward_propagation()
### START CODE HERE ### (approx. 1 line)
grad = backward_propagation(x,theta)
### END CODE HERE ###
### START CODE HERE ### (approx. 1 line)
fenzi = np.linalg.norm(grad-gradapprox) # Step 1'
fenmu = np.linalg.norm(grad)+np.linalg.norm(gradapprox) # Step 2'
difference = fenzi/fenmu # Step 3'
### END CODE HERE ###
if difference < 1e-7:
print ("The gradient is correct!")
else:
print ("The gradient is wrong!")
return difference
高维数据的梯度检验
def forward_propagation_n(X, Y, parameters):
"""
Implements the forward propagation (and computes the cost) presented in Figure 3.
Arguments:
X -- training set for m examples
Y -- labels for m examples
parameters -- python dictionary containing your parameters "W1", "b1", "W2", "b2", "W3", "b3":
W1 -- weight matrix of shape (5, 4)
b1 -- bias vector of shape (5, 1)
W2 -- weight matrix of shape (3, 5)
b2 -- bias vector of shape (3, 1)
W3 -- weight matrix of shape (1, 3)
b3 -- bias vector of shape (1, 1)
Returns:
cost -- the cost function (logistic cost for one example)
"""
# retrieve parameters
m = X.shape[1]
W1 = parameters["W1"]
b1 = parameters["b1"]
W2 = parameters["W2"]
b2 = parameters["b2"]
W3 = parameters["W3"]
b3 = parameters["b3"]
# LINEAR -> RELU -> LINEAR -> RELU -> LINEAR -> SIGMOID
Z1 = np.dot(W1, X) + b1
A1 = relu(Z1)
Z2 = np.dot(W2, A1) + b2
A2 = relu(Z2)
Z3 = np.dot(W3, A2) + b3
A3 = sigmoid(Z3)
# Cost
logprobs = np.multiply(-np.log(A3),Y) + np.multiply(-np.log(1 - A3), 1 - Y)
cost = 1./m * np.sum(logprobs)
cache = (Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3)
return cost, cache
def backward_propagation_n(X, Y, cache):
"""
Implement the backward propagation presented in figure 2.
Arguments:
X -- input datapoint, of shape (input size, 1)
Y -- true "label"
cache -- cache output from forward_propagation_n()
Returns:
gradients -- A dictionary with the gradients of the cost with respect to each parameter, activation and pre-activation variables.
"""
m = X.shape[1]
(Z1, A1, W1, b1, Z2, A2, W2, b2, Z3, A3, W3, b3) = cache
dZ3 = A3 - Y
dW3 = 1./m * np.dot(dZ3, A2.T)
db3 = 1./m * np.sum(dZ3, axis=1, keepdims = True)
dA2 = np.dot(W3.T, dZ3)
dZ2 = np.multiply(dA2, np.int64(A2 > 0))
dW2 = 1./m * np.dot(dZ2, A1.T) * 2
db2 = 1./m * np.sum(dZ2, axis=1, keepdims = True)
dA1 = np.dot(W2.T, dZ2)
dZ1 = np.multiply(dA1, np.int64(A1 > 0))
dW1 = 1./m * np.dot(dZ1, X.T)
db1 = 4./m * np.sum(dZ1, axis=1, keepdims = True)
gradients = {"dZ3": dZ3, "dW3": dW3, "db3": db3,
"dA2": dA2, "dZ2": dZ2, "dW2": dW2, "db2": db2,
"dA1": dA1, "dZ1": dZ1, "dW1": dW1, "db1": db1}
return gradients
梯度检验原理
与1和2中一样,你想将“gradapprox”与通过反向传播计算的梯度进行比较。公式仍然是:
∂ J ∂ θ = lim ε → 0 J ( θ + ε ) − J ( θ − ε ) 2 ε (1) \frac{\partial J}{\partial \theta} = \lim_{\varepsilon \to 0} \frac{J(\theta + \varepsilon) - J(\theta - \varepsilon)}{2 \varepsilon} \tag{1} ∂θ∂J=ε→0lim2εJ(θ+ε)−J(θ−ε)(1)
但是,
θ
\theta
θ不再是标量。 而是一个叫做“参数”的字典。 我们为你实现了一个函数"dictionary_to_vector()
"。它将“参数”字典转换为称为“值”的向量,该向量是通过将所有参数(W1, b1, W2, b2, W3, b3)重塑为向量并将它们串联而获得的。
反函数是“vector_to_dictionary
”,它输出回“parameters”字典。
图2:dictionary_to_vector()和vector_to_dictionary()
你将在 gradient_check_n()中用到这些函数
我们还使用gradients_to_vector()将“gradients”字典转换为向量“grad”。
练习:实现gradient_check_n()。
说明:这是伪代码,可帮助你实现梯度检验。
For each i in num_parameters:
- 计算
J_plus [i]
:
1. 将 θ + \theta^{+} θ+设为np.copy(parameters_values)
2. 将 θ i + \theta^{+}_i θi+ 设为 θ i + + ε \theta^{+}_i + \varepsilon θi++ε
3. 使用forward_propagation_n(x, y, vector_to_dictionary(
θ + \theta^{+} θ+))
计算 J i + J^{+}_i Ji+ - 计算
J_minus [i]
:也是用 θ − \theta^{-} θ− - 计算 g r a d a p p r o x [ i ] = J i + − J i − 2 ε gradapprox[i] = \frac{J^{+}_i - J^{-}_i}{2 \varepsilon} gradapprox[i]=2εJi+−Ji−
因此,你将获得向量gradapprox,其中gradapprox[i]是相对于parameter_values[i]
的梯度的近似值。现在,你可以将此gradapprox向量与反向传播中的梯度向量进行比较。就像一维情况(步骤1’,2’,3’)一样计算:
d
i
f
f
e
r
e
n
c
e
=
∥
g
r
a
d
−
g
r
a
d
a
p
p
r
o
x
∥
2
∥
g
r
a
d
∥
2
+
∥
g
r
a
d
a
p
p
r
o
x
∥
2
(3)
difference = \frac {\| grad - gradapprox \|_2}{\| grad \|_2 + \| gradapprox \|_2 } \tag{3}
difference=∥grad∥2+∥gradapprox∥2∥grad−gradapprox∥2(3)
代码实现
# GRADED FUNCTION: gradient_check_n
def gradient_check_n(parameters, gradients, X, Y, epsilon = 1e-7):
"""
Checks if backward_propagation_n computes correctly the gradient of the cost output by forward_propagation_n
Arguments:
parameters -- python dictionary containing your parameters "W1", "b1", "W2", "b2", "W3", "b3":
grad -- output of backward_propagation_n, contains gradients of the cost with respect to the parameters.
x -- input datapoint, of shape (input size, 1)
y -- true "label"
epsilon -- tiny shift to the input to compute approximated gradient with formula(1)
Returns:
difference -- difference (2) between the approximated gradient and the backward propagation gradient
"""
# Set-up variables
parameters_values, _ = dictionary_to_vector(parameters)
grad = gradients_to_vector(gradients)
num_parameters = parameters_values.shape[0]
J_plus = np.zeros((num_parameters, 1))
J_minus = np.zeros((num_parameters, 1))
gradapprox = np.zeros((num_parameters, 1))
# Compute gradapprox
for i in range(num_parameters):
# Compute J_plus[i]. Inputs: "parameters_values, epsilon". Output = "J_plus[i]".
# "_" is used because the function you have to outputs two parameters but we only care about the first one
### START CODE HERE ### (approx. 3 lines)
thetaplus = np.copy(parameters_values)# Step 1
thetaplus[i] = thetaplus[i]+epsilon # Step 2
J_plus[i],cache_plus = forward_propagation_n(X, Y, vector_to_dictionary(thetaplus)) # Step 3
### END CODE HERE ###
# Compute J_minus[i]. Inputs: "parameters_values, epsilon". Output = "J_minus[i]".
### START CODE HERE ### (approx. 3 lines)
thetaminus = np.copy(parameters_values)# Step 1
thetaminus[i] = thetaminus[i]-epsilon # Step 2
J_minus[i],cache_minus = forward_propagation_n(X, Y, vector_to_dictionary(thetaminus)) # Step 3
### END CODE HERE ###
# Compute gradapprox[i]
### START CODE HERE ### (approx. 1 line)
gradapprox[i] = (J_plus[i]-J_minus[i])/(2*epsilon)
### END CODE HERE ###
# Compare gradapprox to backward propagation gradients by computing difference.
### START CODE HERE ### (approx. 1 line)
fenzi = np.linalg.norm(grad-gradapprox) # Step 1'
fenmu = np.linalg.norm(grad)+np.linalg.norm(gradapprox) # Step 2'
difference = fenzi/fenmu # Step 3'
### END CODE HERE ###
if difference > 1e-7:
print ("\033[93m" + "There is a mistake in the backward propagation! difference = " + str(difference) + "\033[0m")
else:
print ("\033[92m" + "Your backward propagation works perfectly fine! difference = " + str(difference) + "\033[0m")
return difference
差距较大,认为有错误
看起来backward_propagation_n
代码似乎有错误!很好,你已经实现了梯度检验。返回到backward_propagation
并尝试查找/更正错误*(提示:检查dW2和db1)*。如果你已解决问题,请重新运行梯度检验。请记住,如果修改代码,则需要重新执行定义backward_propagation_n()
的单元格。
你可以进行梯度检验来证明你的导数计算的正确吗?即使作业的这一部分没有评分,我们也强烈建议你尝试查找错误并重新运行梯度检验,直到确信实现了正确的反向传播。
注意
- 梯度检验很慢!用 ∂ J ∂ θ ≈ J ( θ + ε ) − J ( θ − ε ) 2 ε \frac{\partial J}{\partial \theta} \approx \frac{J(\theta + \varepsilon) - J(\theta - \varepsilon)}{2 \varepsilon} ∂θ∂J≈2εJ(θ+ε)−J(θ−ε) 逼近梯度在计算上是很耗费资源的。因此,我们不会在训练期间的每次迭代中都进行梯度检验。只需检查几次梯度是否正确。
- 至少如我们介绍的那样,梯度检验不适用于dropout。通常,你将运行不带dropout的梯度检验算法以确保你的backprop是正确的,然后添加dropout。
Nice!现在你可以确信你用于欺诈检测的深度学习模型可以正常工作!甚至可以用它来说服你的CEO。 😃
你在此笔记本中应记住的内容:
- 梯度检验可验证反向传播的梯度与梯度的数值近似值之间的接近度(使用正向传播进行计算)。
- 梯度检验很慢,因此我们不会在每次训练中都运行它。通常,你仅需确保其代码正确即可运行它,然后将其关闭并将backprop用于实际的学习过程。