- 目的:找到损失函数的最小值;方法:梯度下降
- z = w T x + b z = w^Tx + b z=wTx+b
-
y
^
=
a
=
σ
(
z
)
\hat{y} =a= \sigma(z)
y^=a=σ(z)
损失函数
- 单个训练样本的损失函数:
L ( y ^ , y ) = − ( y log y ^ ) − ( 1 − y ) log ( 1 − y ^ ) L(\hat{y},y) = -(y\log\hat{y})-(1-y)\log(1-\hat{y}) L(y^,y)=−(ylogy^)−(1−y)log(1−y^)
L ( a , y ) = − ( y log a ) − ( 1 − y ) log ( 1 − a ) L(a,y) = -(y\log{a})-(1-y)\log(1-a) L(a,y)=−(yloga)−(1−y)log(1−a)
- 全体训练样本的损失函数:
J ( W , b ) = 1 m ∑ i = 1 m L ( y ^ ( i ) , y ( i ) ) J(W,b) = \frac{1}{m}\sum_{i=1}^mL(\hat{y}^{(i)},y^{(i)}) J(W,b)=m1i=1∑mL(y^(i),y(i))
梯度下降过程计算(注意字母大小写)
- 单个训练样本,参数w和b的更新公式为:
w : = w − α ∂ J ( w , b ) ∂ w , b : = b − α ∂ J ( w , b ) ∂ b w := w - \alpha\frac{\partial J(w, b)}{\partial w},b := b - \alpha\frac{\partial J(w, b)}{\partial b} w:=w−α∂w∂J(w,b),b:=b−α∂b∂J(w,b) - 全体训练样本,参数
W
W
W和
b
b
b的更新公式为:
W : = W − α d W , b : = b − α d b W := W - \alpha dW,b := b - \alpha db W:=W−αdW,b:=b−αdb - 计算
J
J
J关于
z
z
z的导数:
∂ J ∂ a = − y a + 1 − y 1 − a \frac{\partial J}{\partial a}=-\frac{y}{a} + \frac{1-y}{1-a} ∂a∂J=−ay+1−a1−y
a = σ ( z ) = 1 1 + e − z ⟹ σ ( z ) ′ = σ ( z ) ( 1 − σ ( z ) ) ⟹ ∂ a ∂ z = a ( 1 − a ) a=\sigma(z) = \frac{1}{1+e^{-z}} \implies \sigma(z)^\prime=\sigma(z)(1-\sigma(z)) \implies \frac{\partial a}{\partial z} =a(1-a) a=σ(z)=1+e−z1⟹σ(z)′=σ(z)(1−σ(z))⟹∂z∂a=a(1−a)
⟹ d z = ∂ J ∂ z = ∂ J ∂ a ∗ ∂ a ∂ z = ( − y a + 1 − y 1 − a ) ∗ a ( 1 − a ) = a − y \implies dz =\frac{\partial J}{\partial z}= \frac{\partial J}{\partial a}*\frac{\partial a}{\partial z} = (-\frac{y}{a} + \frac{1-y}{1-a}) * a(1-a) = a-y ⟹dz=∂z∂J=∂a∂J∗∂z∂a=(−ay+1−a1−y)∗a(1−a)=a−y
- 计算
J
J
J关于
W
W
W的导数:
d w = ∂ J ∂ w = ∂ J ∂ a ∗ ∂ a ∂ z ∗ ∂ z ∂ w = d z ∗ x = x d z ⟹ d W = ∂ J ∂ W = 1 m X d Z T dw=\frac{\partial J}{\partial w}= \frac{\partial J}{\partial a}*\frac{\partial a}{\partial z} * \frac{\partial z}{\partial w}=dz*x=xdz\implies dW=\frac{\partial J}{\partial W}=\frac{1}{m}XdZ^T dw=∂w∂J=∂a∂J∗∂z∂a∗∂w∂z=dz∗x=xdz⟹dW=∂W∂J=m1XdZT - 计算
J
J
J关于
b
b
b的导数:
d b = ∂ J ∂ b = ∂ J ∂ a ∗ ∂ a ∂ z ∗ ∂ z ∂ b = d z ∗ 1 = d z ⟹ d b = 1 m n p . s u m ( d Z ) db=\frac{\partial J}{\partial b}= \frac{\partial J}{\partial a}*\frac{\partial a}{\partial z} * \frac{\partial z}{\partial b}=dz*1=dz\implies db=\frac{1}{m}np.sum(dZ) db=∂b∂J=∂a∂J∗∂z∂a∗∂b∂z=dz∗1=dz⟹db=m1np.sum(dZ)