Class1-Week4-Deep Neural Network

Compute Process

Forward Propagation

Layer-l:

• Input: $A^{[l-1]}$
• Compute Process:
  $$Z^{[l]}=W^{[l]}{A}^{[l-1]}+b^{[l]}$$
  $$A^{[l]}=g(Z^{[l]})$$
• Output: $A^{[l]}$
• Cache: $Z^{[l]},W^{[l]},b^{[l]}$

Backward Propagation

Layer-l:

• Input: $d{A}^{[l]}$
• Compute Process:
  $$d{Z}^{[l]}=d{A}^{[l]}\ast g^{\prime }(Z^{[l]})$$
  $$d{W}^{[l]}=\frac{1}{m}\ast d{Z}^{[l]}{A}^{[l-1]T}$$
  $$d{b}^{[l]}=\frac{1}{m}\ast np.sum(d{Z}^{[l]},axis=1,keepdims=True)$$
  $$d{A}^{[l-1]}=W^{[l]T}d{Z}^{[l]}$$
• Output: $d{A}^{[l-1]}$
• Update:
  $$W^{[l]}=W^{[l]}-\alpha d{W}^{[l]}$$
  $$b^{[l]}=b^{[l]}-\alpha d{b}^{[l]}$$

Matrix Dimensions

Layer-l:

$$\begin{array}{rl}d{W}^{[l]}=W^{[l]}& :(n^{[l]},n^{[l-1]})\\ d{b}^{[l]}=b^{[l]}& :(n^{[l]},1)\\ d{Z}^{[l]}=Z^{[l]}& :(n^{[l]},m)\\ d{A}^{[l]}=A^{[l]}& :(n^{[l]},m)\end{array}$$

---

Parameters vs Hyperparameters

Defination

In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. By contrast, the values of other parameters are derived via training.
1. Parameters: W, b
2. Hyperparameters:
- Learning_rate $\alpha$ – we can set a proper learning rate by drawing the relationship graph between iterations and cost in different learning rate.
- Iteration_numbers
- Network architecture
- Activation functions
- …

Tune for Hyperparameters