Backpropagation

1. Loss function content:
   $$L(\theta )=\sum _i{C}_i^n(\theta )=\sum _i{o}_i^2-{\widehat{o_i}}^2$$
   

2. how to calculate the partial value of partial of var

$$\frac{\partial C_i}{\partial \theta }=\frac{\partial o_i}{\partial \theta }\ast 2o_i$$

3. for example, we calculate the partial of w11 in the level n-1.

$$\frac{\partial o_i}{\partial w_{11}^{n-1}}=\frac{\partial o_i}{\partial y_1^{n-1}}\frac{\partial y_1^{n-1}}{\partial w_{11}^{n-1}}=\frac{\partial o_i}{\partial y_1^{n-1}}{x}_1$$

4. backward pass: we convert the question to calculate

$$\frac{\partial o_i}{\partial y_1^{n-1}}=\frac{\partial o_i}{\partial x_1^n}\frac{\partial x_1^n}{\partial y_1^{n-1}}=\frac{\partial o_i}{\partial x_1^n}{\varphi }^{\prime }(y_1^{n-1})$$

$$\frac{\partial o_i}{\partial x_1^n}=\frac{\partial o_i}{\partial y_1^n}{w}_{11}^n+\frac{\partial o_i}{\partial y_2^n}{w}_{12}^n+\frac{\partial o_i}{\partial y_3^n}{w}_{13}^n$$

Therefore, we have

$$\left[\begin{array}{c}\frac{\partial o_i}{\partial y_1^{n-1}}\\ \frac{\partial o_i}{\partial y_2^{n-1}}\\ \frac{\partial o_i}{\partial y_3^{n-1}}\end{array}\right]=\left[\begin{array}{ccc}{\varphi }^{\prime }(y_1^{n-1})& 0& 0\\ 0& {\varphi }^{\prime }(y_2^{n-1})& 0\\ 0& 0& {\varphi }^{\prime }(y_3^{n-1})\end{array}\right]{\left[\begin{array}{c}{w}_{11}{w}_{12}{w}_{13}\\ w_{21}{w}_{22}{w}_{23}\\ w_{31}{w}_{32}{w}_{33}\end{array}\right]}^n\left[\begin{array}{c}\frac{\partial o_i}{\partial y_1^n}\\ \frac{\partial o_i}{\partial y_2^n}\\ \frac{\partial o_i}{\partial y_3^n}\end{array}\right]$$