吴恩达神经网络与深度学习——深度神经网络

深度神经网络

符号

l：层数
l = 4
n^[l]:每一次的单元数
n^[1] = 5 n^[2] = 5 n^[3] = 3 n^[4] = 1
a^[l]:每一次的激活函数
a^[l] = g^[l](z^[l])
w^[l]:每一次的权值
b^[l]:每一次的偏置

前向传播

x
z^[1] = w^[1]x + b^[1]
a^[1] = g^[1](z^[1])
z^[2] = w^[2]a^[1] + b^[2]
a^[2] = g^[2](z^[2])
z^[3] = w^[3]a^[2] + b^[3]
a^[3] = g^[3](z^[3])
z^[4] = w^[4]a^[3] + b^[4]
a^[4] = g^[4](z^[4])

for l =1 to 4
    z^[l] = w^[l]a^[l-1] + b^[l]
    a^[l] = g^[l](z^[l])

#m个样本向量化
Z^[1] = W^[1]A^[0] + b^[1]      #  X=A^[0]
A^[1] = g^[1](Z^[1])
Z^[2] = W^[2]A^[1]+ b^[2]
A^[2] = g^[2](z^[2])
Z^[3] = W^[3]A^[2] + b^[3]
A^[3] = g^[3](Z^[3])
Z^[4] = W^[4]A^[3] + b^[4]
A^[4] = g^[4](Z^[4])

for l = 1 to 4
	    Z^[l] = w^[l]A^[l-1] + b^[l]
	    A^[l] = g^[l](Z^[l])

矩阵维度

n^[0] = 2
n^[1] = 3
n^[2] = 5
n^[3] = 4
n^[4] = 2
n^[4] = 1
z^[1]        =      w^[1]               x            +        b^[1]
(3,1)               (3,2)             (2,1)                   (3,1)
(n^[1],1)        (n^[1],n^[0])       (n^[0],1)              (n^[1],1)
a^[1] = g^[1](z^[1])
(3,1)         (3,1)
(n^[1],1)   (n^[1],1)
z^[2]        =      w^[2]             a^[1]             +     b^[1]
(5,1)               (5,3)             (3,1)                   (5,1)
(n^[2],1)        (n^[2],n^[1])       (n^[1],1)              (n^[2],1)
a^[2] = g^[2](z^[2])
(5,1)         (5,1) 
(n^[2],1)   (n^[2],1)  
z^[3]        =      w^[3]             a^[2]             +     b^[3]
(4,1)               (4,5)             (5,1)                   (4,1)
(n^[3],1)        (n^[3],n^[2])       (n^[2],1)              (n^[3],1)
a^[3] = g^[3](z^[3])
(4,1)         (4,1)  
(n^[3],1)   (n^[3],1)  
z^[4]        =      w^[4]             a^[3]             +     b^[4]
(2,1)               (2,4)             (4,1)                   (2,1)
(n^[4],1)        (n^[4],n^[3])       (n^[3],1)              (n^[4],1)
a^[4] = g^[4](z^[4])
(2,1)         (2,1)        
(n^[4],1)    (n^[4],1)    
z^[5]        =      w^[5]             a^[4]             +     b^[5]
(1,1)               (1,2)             (2,1)                   (1,1)
(n^[5],1)        (n^[5],n^[4])       (n^[4],1)              (n^[5],1)
a^[5] = g^[5](z^[5])
(1,1)         (1,1)        
(n^[5],1)    (n^[5],1)    

for l = 1 to 5
	    z^[l]     =   w^[l]            a^[l-1]         +       b^[l]
	    (n^[l],1)    (n^[l],n^[l-1])  (n^[l-1],1)            (n^[l],1)
	    a^[l] = g^[l](z^[l])
	    (n^[l],1)    (n^[l],1) 

m个样本

Z^[1]        =      W^[1]               X            +        b^[1]
(3,m)               (3,2)             (2,m)                   (3,1)
(n^[1],m)        (n^[1],n^[0])       (n^[0],m)              (n^[1],1)
A^[1] = g^[1](Z^[1])
(3,m)         (3,m)
(n^[1],m)   (n^[1],m)
Z^[2]        =      W^[2]             A^[1]             +     b^[1]
(5,m)               (5,3)             (3,m)                   (5,1)
(n^[2],m)        (n^[2],n^[1])       (n^[1],m)              (n^[2],1)
A^[2] = g^[2](Z^[2])
(5,m)         (5,m) 
(n^[2],m)   (n^[2],m)  
Z^[3]        =      W^[3]             A^[2]             +     b^[3]
(4,m)               (4,5)             (5,m)                   (4,1)
(n^[3],m)        (n^[3],n^[2])       (n^[2],m)              (n^[3],1)
A^[3] = g^[3](Z^[3])
(4,m)         (4,m)  
(n^[3],m)   (n^[3],m)  
Z^[4]        =      W^[4]             A^[3]             +     b^[4]
(2,m)               (2,4)             (4,m)                   (2,1)
(n^[4],m)        (n^[4],n^[3])       (n^[3],m)              (n^[4],1)
A^[4] = g^[4](Z^[4])
(2,m)         (2,m)        
(n^[4],m)    (n^[4],m)    
Z^[5]        =      W^[5]             A^[4]             +     b^[5]
(1,m)               (1,2)             (2,m)                   (1,1)
(n^[5],m)        (n^[5],n^[4])       (n^[4],m)              (n^[5],1)
A^[5] = g^[5](Z^[5])
(1,m)         (1,m)        
(n^[5],m)    (n^[5],m)    

for l = 1 to 4
	    Z^[l]    =    w^[l]              A^[l-1]   +     b^[l]
	    (n^[l],m)    (n^[l],n^[l-1])   (n^[l-1],m)     (n^[l],1)
	    A^[l] = g^[l](Z^[l])
      (n^[l],m)     (n^[l],m)

为什么使用深层表示

深度神经网络有很多的隐层，较早的前几层能学习一些低层次的简单特征，后几层就能将简单的特征结合起来去探测更加复杂的东西

搭建深层神经网络块

第l层参数： w^[l],b^[l]
前向传播： 输入 a^[l-1] 输出 a^[l]  存储 z^[l]
					z^[l] = w^[l]a^[l-1] + b^[l]
					a^[l] = g^[l](z^[l])
反向传播： 输入 da^[l]   输出da^[l-1]  dw^[l] db^[l]
					前向传播存储的z^[l] 					
					

正向传播和反向传播

从a^[0]开始，也就是x，经过一系列正向传播计算得到yhat,之后再用输出值计算实现反向传播，得到所有的导数项，w，b也在每一层更新

编程细节:将z^[l],w^[l],b^[l]存储

前向和反向传播

前向传播

前向传播： 输入 a^[l-1] 输出 a^[l]  存储 z^[l]
					z^[l] = w^[l]a^[l-1] + b^[l]
					a^[l] = g^[l](z^[l])
		向量化：
					Z^[l] = W^[l]A^[l-1] + b^[l]
					A^[l] = g^[l](Z^[l])

反向传播

反向传播： 输入 da^[l]   输出da^[l-1]  dw^[l] db^[l]
					dz^[l] = da^[l]*g'^[l](z^[l])
					dw^[l] = dz^[l]a^[l-1]
					db^[l] = dz^[l]
					da^[l-1] = w^[l]Tdz^[l]
					dz^[l] = w^[l+1]Tdz^[l+1]*g'^[l](z^[l])
		向量化：
					dZ^[l] = dA^[l]*g'^[l](Z^[l])
					dW^[l] =(1/m)dZ^[l]A^[l-1]T
					db^[l] = (1/m)np.sum(dZ^[l],axis = 1,keepdims = True)
					dA^[l-1] = W^[l]TdZ^[l]
					dZ^[l] = W^[l+1]TdZ^[l+1]*g'^[l](Z^[l])

参数和超参数

参数： w^[1],b^[1],w^[2],b^[2]...
超参数：
			学习率：alpha
			循环下降法的迭代次数：iteration
			隐藏层数：l
			隐藏单元数：n^[1],n^[2]...
			激活函数：sigmoid ,relu, tanh
			

和大脑的关系