第三周:浅层神经网络
本文是神经网络和深度学习课程的笔记。
3.0 重要的结论:
- 参数更新:
- 单个样本
- 第k层的输出: a [ z ] = g ( z [ k ] ) , z [ k ] = w [ k ] a [ k − 1 ] + b [ k ] a^{[z]}=g(z^{[k]}),z^{[k]}=w^{[k]}a^{[k-1]}+b^{[k]} a[z]=g(z[k]),z[k]=w[k]a[k−1]+b[k],
其中 w [ k ] . s h a p e = ( n [ k ] , n [ k − 1 ] ) w^{[k]}.shape=(n^{[k]},n^{[k-1]}) w[k].shape=(n[k],n[k−1])
b [ k ] . s h a p e = ( n [ k ] , 1 ) b^{[k]}.shape=(n^{[k]},1) b[k].shape=(n[k],1)
z [ k ] . s h a p e = ( n [ k ] , 1 ) z^{[k]}.shape=(n^{[k]},1) z[k].shape=(n[k],1)
a [ k ] . s h a p e = ( n [ k ] , 1 ) a^{[k]}.shape=(n^{[k]},1) a[k].shape=(n[k],1) - 参数更新:
- d z [ k ] = ∂ L O S S ∂ z [ k + 1 ] ∂ z [ k + 1 ] ∂ a [ k ] ∗ ∂ a [ k ] ∂ z [ k ] = w [ k + 1 ] , T d z [ k + 1 ] ∗ g ′ ( z [ k ] ) dz^{[k]}= \frac{\partial LOSS}{\partial z^{[k+1]}} \frac{\partial z^{[k+1]}}{\partial a^{[k]} } * \frac{\partial a^{[k]}}{\partial z^{[k]}}=w^{[k+1],T}dz^{[k+1]} *g'(z^{[k]}) dz[k]=∂z[k+1]∂LOSS∂a[k]∂z[k+1]∗∂z[k]∂a[k]=w[k+1],Tdz[k+1]∗g′(z[k]),因为 a [ k ] = g ( z [ k ] ) , z [ k + 1 ] = w [ k + 1 ] a [ k ] + b [ k + 1 ] a^{[k]}=g(z^{[k]}),z^{[k+1]}=w^{[k+1]}a^{[k]}+b^{[k+1]} a[k]=g(z[k]),z[k+1]=w[k+1]a[k]+b[k+1] *表示对应元素相乘。
- d w [ k ] = d z [ k ] a [ k − 1 ] , T dw^{[k]}=dz^{[k]}a^{[k-1],T} dw[k]=dz[k]a[k−1],T
- d b [ k ] = d z [ k ] db^{[k]}=dz^{[k]} db[k]=dz[k]
- 具体实现
d z [ 2 ] = a [ 2 ] − y dz^{[2]}=a^{[2]}-y dz[2]=a[2]−y 鉴于g(z)为sigmoid
d W [ 2 ] = d z [ 2 ] a [ 1 ] , T dW^{[2]}=dz^{[2]}a^{[1],T} dW[2]=dz[2]a[1],T
d b [ 2 ] = d z [ 2 ] db^{[2]}=dz^{[2]} db[2]=dz[2]
d z [ 1 ] = W [ 2 ] , T d z [ 2 ] ∗ g [ 1 ] ′ ( z [ 1 ] ) dz^{[1]}=W^{[2],T}dz^{[2]}*g^{[1]'}(z^{[1]}) dz[1]=W[2],Tdz[2]∗g[1]′(z[1])
d W [ 1 ] = d z [ 1 ] a [ 0 ] , T = d z [ 1 ] x T dW^{[1]}=dz^{[1]}a^{[0],T}=dz^{[1]}x^{T} dW[1]=dz[1]a[0],T=dz[1]xT
d b [ 1 ] = d z [ 1 ] db^{[1]}=dz^{[1]} db[1]=dz[1]
- 第k层的输出: a [ z ] = g ( z [ k ] ) , z [ k ] = w [ k ] a [ k − 1 ] + b [ k ] a^{[z]}=g(z^{[k]}),z^{[k]}=w^{[k]}a^{[k-1]}+b^{[k]} a[z]=g(z[k]),z[k]=w[k]a[k−1]+b[k],
- 多个样本
- 通过将多个样本横着挤在一起: [ x ( 1 ) , . . . x ( n ) ] [x^{(1)},...x^{(n)}] [x(1),...x(n)] 引入 X , Y , A [ n ] , Z [ n ] X,Y,A^{[n]},Z^{[n]} X,Y,A[n],Z[n]
从而具体实现为:
d Z [ 2 ] = A [ 2 ] − Y dZ^{[2]}=A^{[2]}-Y dZ[2]=A[2]−Y 鉴于g(z)为sigmoid
d W [ 2 ] = 1 m d Z [ 2 ] A [ 1 ] , T dW^{[2]}=\frac{1}{m}dZ^{[2]}A^{[1],T} dW[2]=m1dZ[2]A[1],T 因为Cost Function里面有一个 1 m \frac{1}{m} m1
d b [ 2 ] = 1 m n p . s u m ( d Z [ 2 ] , a x i s = 1 , k e e p d i m s = T r u e ) db^{[2]}=\frac{1}{m} np.sum(dZ^{[2]},axis=1,keepdims=True) db[2]=
- 通过将多个样本横着挤在一起: [ x ( 1 ) , . . . x ( n ) ] [x^{(1)},...x^{(n)}] [x(1),...x(n)] 引入 X , Y , A [ n ] , Z [ n ] X,Y,A^{[n]},Z^{[n]} X,Y,A[n],Z[n]