Shallow Neural Network Week 3

Single Sample

Symbols

$X = \begin{pmatrix} x_1 \\ \vdots \\ x_{n _{x}} \end{pmatrix}, Y = \begin{pmatrix} y_1 \\ \vdots \\ y_{n _{y}} \end{pmatrix},$
$Z ^{[l]} = \begin{pmatrix} z ^{[l]} _1 \\ \vdots \\ z ^{[l]} _{ n_{l}} \end{pmatrix}, 1 \le l \le L$
$A ^{[l]} = \begin{pmatrix} a ^{[l]} _1 \\ \vdots \\ a ^{[l]} _{ n_{l}} \end{pmatrix}, \tilde A ^{[l]} = \begin{pmatrix} a ^{[l]} _0 \\ a ^{[l]} _1 \\ \vdots \\ a ^{[l]} _{ n_{l}} \end{pmatrix} = \begin{pmatrix} 1 \\ A ^{[l]} \end{pmatrix},0 \le l \le L$
$W ^{[l]} = \begin{pmatrix} w ^{[l]} _{ij} \end{pmatrix} _{ n_{l} \times n_{ l - 1 }} , w' ^{[l]} = \begin{pmatrix} w ^{[l]} _{1, 0} \\ \vdots \\ w ^{[l]} _{n_{l}, 0} \end{pmatrix}, \tilde W ^{[l]} = \begin{pmatrix} w' ^{[l]} & W ^{[l]} \end{pmatrix}, 1 \le l \lt L$

Neural Network Architecture

$X = A ^{[0]} \rightarrow Z ^{[1]} \rightarrow A ^{[1]} \rightarrow \cdots \rightarrow Z ^{[L]} \rightarrow A ^{[L]} = \hat {Y}$

Loss Function

$z ^{[l]} _{i} = \sum \limits_{j = 0} ^{ n_{l - 1} } w^{[l]} _{ij} \tilde a^{[l - 1]} _{j}, 1 \le i \le n_{l}, 1 \le l \le L$
即 $Z ^{l} = W^{[l]} \tilde A ^{[l - 1]}, 1 \le l \le L$
$a ^{[l]} _{i} = g \left (z ^{[l]} _{i} \right ), 1 \le i \le n_{l}, 1 \le l \le L$
即 $A ^{[l]} = g \left (Z ^{[l]} \right ), 1 \le l \le L$
$\operatorname {loss} (X, Y) = - \sum \limits_{i = 1} ^{n_{y}} \left [ y_{i} \ln \hat y_{i} + \left ( 1 - y_{i} \right ) \ln \left ( 1 - \hat y_{i} \right ) \right ]$

公式

$\dfrac {\partial}{ \partial z ^{[L]} _{i} } \operatorname {loss} (X, Y) = \dfrac {\operatorname {d} \hat y_{i}}{ \operatorname {d} z^{[L]} _{i}} \cdot \dfrac {\partial}{ \partial \hat y_{i} } \operatorname {loss} (X, Y)$
$= - g ' \left (z ^{[L]} \right ) \left [ y_{i} \cdot \dfrac {1}{\hat y_{i} } - \left ( 1 - y_{i} \right ) \cdot \dfrac {1}{1 - \hat y_{i} } \right ]$
$= - \hat y_{i} \left ( 1 - \hat y_{i} \right ) \left [ y_{i} \cdot \dfrac {1}{\hat y_{i} } - \left ( 1 - y_{i} \right ) \cdot \dfrac {1}{1 - \hat y_{i} } \right ]$
$= \left ( 1 - y_{i} \right ) \hat y_{i} - y_{i} \left ( 1 - \hat y_{i} \right )$
$= \hat y_{i} - y_{i}, 1 \le i \le n_{L}$

$\dfrac {\partial}{ \partial z ^{[l]} _{j} } \operatorname {loss} (X, Y) = \sum \limits_{i = 1} ^{ n_{l + 1}} \dfrac {\partial z ^{[l + 1]} _{i}}{ \partial z ^{[l]} _{j} } \cdot \dfrac {\partial}{ \partial z ^{[l + 1]} _{i} } \operatorname {loss} (X, Y)$
$= \sum \limits_{i = 1} ^{ n_{l + 1}} g ' \left ( z ^{[l]} _{j} \right ) w^{[l]} _{ij} \cdot \dfrac {\partial}{ \partial z ^{[l + 1]} _{i} } \operatorname {loss} (X, Y)$
$= g ' \left ( z ^{[l]} _{j} \right ) \sum \limits_{i = 1} ^{ n_{l + 1}} w^{[l]} _{ij} \cdot \dfrac {\partial}{ \partial z ^{[l + 1]} _{i} } \operatorname {loss} (X, Y), 1 \le j \le s _{l}, 1 \le l \lt L$
因此
$\dfrac {\partial}{ \partial Z ^{[l]} } \operatorname {loss} (X, Y) = \begin{cases} A ^{[L]} - Y, l = L \\ g ' \left ( Z ^{[l]} \right ) \ .*\ \left ( \left ( W ^{[l + 1]} \right ) ^{\intercal} \dfrac {\partial}{ \partial Z ^{[l + 1]} } \operatorname {loss} (X, Y) \right ), 1 \le l \lt L \end{cases}$
where .* is element-wise product.

$\dfrac {\partial}{ \partial w ^{[l]} _{ij} } \operatorname {loss} (X, Y) = \dfrac {\partial}{ \partial z ^{[l]} _{i} } \operatorname {loss} (X, Y) \cdot \tilde a ^{[l - 1]} _{j}, 1 \le i \le s _{l + 1}, 0 \le j \le s _{l}, 1 \le l \le L$
因此
$\dfrac {\partial}{ \partial \tilde W ^{[l]} } \operatorname {loss} (X, Y) = \dfrac {\partial}{ \partial Z ^{[l]} } \operatorname {loss} (X, Y) \cdot { \tilde A ^{[l - 1]}} ^{\intercal}, 1 \le l \le L$

Multiple Samples

Symbols

$\mathbf {X} = \begin{pmatrix} X ^{(1)}, & \cdots, & X ^{(m)} \end{pmatrix},$
$\mathbf {Y} = \begin{pmatrix} Y ^{(1)}, & \cdots, & Y ^{(m)} \end{pmatrix},$
$\mathbf {Z} ^{[l]} = \begin{pmatrix} Z ^{[l] (1)}, & \cdots, & Z ^{[l] (m)} \end{pmatrix}, 1 \le l \le L$
$\mathbf {A} ^{[l]} = \begin{pmatrix} A^{[l](1)}, & \cdots, & A^{[l](m)} \end{pmatrix},0 \le l \le L$
$\mathbf { \tilde A} ^{[l]} = \begin{pmatrix} \tilde A ^{[l](1)}, & \cdots, & \tilde A ^{[l](m)} \end{pmatrix},0 \le l \le L$
$\partial \mathbf {Z} ^{[l]} = \begin{pmatrix} \dfrac {\partial}{ \partial Z ^{[l]} } \operatorname {loss} \left (X^{(1)}, Y^{(1)} \right ), & \cdots, & \dfrac {\partial}{ \partial Z ^{[l]} } \operatorname {loss} \left (X^{(m)}, Y^{(m)} \right ) \end{pmatrix} _{ n_{l} \times m}, 1 \le l \le L$

Cost Function

$\operatorname {cost} (\mathbf {X}, \mathbf {Y}) = \dfrac {1} {m} \sum \limits_{i = 1} ^{m} \operatorname {loss} \left (X^{(i)}, Y^{(i)} \right )$

公式

$\mathbf {Z} ^{[l]} = W^{[l]} \mathbf { \tilde A} ^{[l - 1]} , 1 \le l \lt L$
$\mathbf {A} ^{[l]} = g \left ( \mathbf {Z} ^{[l]} \right ) , 1 \le l \le L$
$g' \left ( \mathbf {Z} ^{[l]} \right ) = \mathbf {A} ^{[l]} \ .* \ \left ( 1_{ n_{l} \times m} - \mathbf {A} ^{[l]} \right ), 1 \le l \le L$

$\partial \mathbf {Z} ^{[l]} = \begin{cases} \mathbf {A} ^{[L]} - \mathbf {Y}, l = L \\ g ' \left ( \mathbf {Z} ^{[l]} \right ) \ .*\ \left ( \left ( W ^{[l + 1]} \right ) ^{\intercal} \cdot \partial \mathbf {Z} ^{[l + 1]} \right ), 1 \le l \lt L \end{cases}$
$\dfrac {\partial}{ \partial \tilde W ^{[l]} } \operatorname {cost} (\mathbf {X}, \mathbf {Y}) = \dfrac {1} {m} \partial \mathbf {Z} ^{[l]} \cdot { \mathbf { \tilde A} ^{[l - 1]}} ^{\intercal}, 1 \le l \le L$