学习笔记: Deep Learning Specialization: Neural Networks and Deep Learning - 基本套路
1. Forward Propagation (前向算损失)
Z
[
l
]
Z^{[l]}
Z[l]: Linear function
Z
[
l
]
=
W
[
l
]
A
[
l
−
1
]
+
b
[
l
]
Z^{[l]} = W^{[l]} A^{[l-1]} + b^{[l]}
Z[l]=W[l]A[l−1]+b[l]
A
[
l
]
=
g
[
l
]
(
Z
[
l
]
)
A^{[l]} = g^{[l]}(Z^{[l]})
A[l]=g[l](Z[l])
A
[
l
A^{[l}
A[l是Activation函数的结果,注意
A
[
0
]
=
X
A^{[0]} = X
A[0]=X,
X
X
X是输入。
1.1 激活函数 g [ l ] g^{[l]} g[l]
课程中一共介绍了三种:
- Sigmoid: 一般只用在二分类的输出层
sigmoid ( x ) = 1 1 + e − x \text{sigmoid}(x) = \frac{1}{1 + e^{-x}} sigmoid(x)=1+e−x1
Tanh: 在0点附近的梯度比sigmoid大,作为激活函数几乎完全比sigmoid好
tanh ( x ) = e x − e − x e x + e − x \text{tanh}(x) = \frac{e^x - e{-x}}{e^x + e^{-x}} tanh(x)=ex+e−xex−e−x
Relu: 大部分情况下应该选用(sigmoid/tanh在x较大/较小时,学习速度会大大下降)
Relu(z) = max ( 0 , z ) \text{Relu(z)} = \max(0, z) Relu(z)=max(0,z)
2. Cost
适用于二分类
J
=
−
1
m
∑
i
=
0
m
(
y
(
i
)
log
(
a
[
2
]
(
i
)
)
+
(
1
−
y
(
i
)
)
log
(
1
−
a
[
2
]
(
i
)
)
)
J = - \frac{1}{m} \sum\limits_{i = 0}^{m} \large{(} \small y^{(i)}\log\left(a^{[2] (i)}\right) + (1-y^{(i)})\log\left(1- a^{[2] (i)}\right) \large{)} \small
J=−m1i=0∑m(y(i)log(a[2](i))+(1−y(i))log(1−a[2](i)))
代码比公式简单
logprobs = Y * np.log(A_l) + (1 - Y) * np.log(1 - A_l)
cost = - np.sum(logprobs) / m
这里假定了最后一层是 l l l
3. Back Propagation (后向算梯度)
设最后一层是 l l l, i i i是中间任意一层
d
W
[
l
]
=
∂
L
∂
W
[
l
]
=
1
m
d
Z
[
l
]
A
[
l
−
1
]
T
dW^{[l]} = \frac{\partial \mathcal{L} }{\partial W^{[l]}} = \frac{1}{m} dZ^{[l]} A^{[l-1] T}
dW[l]=∂W[l]∂L=m1dZ[l]A[l−1]T
d
b
[
l
]
=
∂
L
∂
b
[
l
]
=
1
m
∑
i
=
1
m
d
Z
[
l
]
(
i
)
db^{[l]} = \frac{\partial \mathcal{L} }{\partial b^{[l]}} = \frac{1}{m} \sum_{i = 1}^{m} dZ^{[l](i)}
db[l]=∂b[l]∂L=m1i=1∑mdZ[l](i)
d
A
[
l
−
1
]
=
∂
L
∂
A
[
l
−
1
]
=
W
[
l
]
T
d
Z
[
l
]
dA^{[l-1]} = \frac{\partial \mathcal{L} }{\partial A^{[l-1]}} = W^{[l] T} dZ^{[l]}
dA[l−1]=∂A[l−1]∂L=W[l]TdZ[l]
考虑到最后一层在二分类时使用的是sigmoid激活函数,也就是
d
Z
[
l
]
=
A
[
l
]
−
Y
dZ^{[l]} = A^{[l]} - Y
dZ[l]=A[l]−Y
4. 初始化
随机即可 W W W部分不能初始化为0,初始化为零后会使网络退化至线性。
-
W
W
W随机初始化至一个比较小的值,比如
np.random.randn(n_h, n_x) * 0.01
- b b b初始化为0
在系列课第二门中会专门讲到这个问题,这里了解到这些也足够了。