LaTeX / Markdown 书写公式

Here’s the regularized cross-entropy:

$$
\begin{aligned} 
C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln (1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2. \tag{85}
\end{aligned}
$$

$$\begin{array}{r}C=-\frac{1}{n}\sum _{xj}\left[y_j\ln a_j^L+(1-y_j)\ln (1-a_j^L)\right]+\frac{\lambda }{2n}\sum _w{w}^2.\end{array}\text{\hspace{1em}(85)}$$

It’s possible to regularize other cost functions, such as the quadratic cost. This can be done in a similar way:

$$
\begin{aligned} 
C = \frac{1}{2n} \sum_x \|y-a^L\|^2 + \frac{\lambda}{2n} \sum_w w^2. \tag{86}
\end{aligned}
$$

$$\begin{array}{r}C=\frac{1}{2n}\sum _x\Vert y-a^L{\Vert }^2+\frac{\lambda }{2n}\sum _w{w}^2.\end{array}\text{\hspace{1em}(86)}$$

In both cases we can write the regularized cost function as

$$
\begin{aligned}
C = C_0 + \frac{\lambda}{2n} \sum_w w^2. \tag{87}
\end{aligned}
$$

$$\begin{array}{r}C=C_0+\frac{\lambda }{2n}\sum _w{w}^2.\end{array}\text{\hspace{1em}(87)}$$

where $C_0$ is the original, unregularized cost function.

Taking the partial derivatives of Equation (87) gives

$$
\begin{aligned} 
\frac{\partial C}{\partial w} & = \frac{\partial C_0}{\partial w} + \frac{\lambda}{n} w \tag{88}\\ 
\end{aligned}
$$

$$
\begin{aligned} 
\frac{\partial C}{\partial b} & = \frac{\partial C_0}{\partial b}. \tag{89}
\end{aligned}
$$

$$\begin{array}{rl}\frac{\partial C}{\partial w}& =\frac{\partial C_0}{\partial w}+\frac{\lambda }{n}w\end{array}\text{\hspace{1em}(88)}$$

$$\begin{array}{rl}\frac{\partial C}{\partial b}& =\frac{\partial C_0}{\partial b}.\end{array}\text{\hspace{1em}(89)}$$

The $\partial C_0/\partial w$ and $\partial C_0/\partial b$ terms can be computed using backpropagation.

DenseNet.

$$
\begin{aligned}
x_{1} &= w_{1} * x_{0} \\
x_{2} &= w_{2} * [x_{0}, x_{1}] \\ 
\vdots \\
x_{k} &= w_{k} * [x_{0}, x_{1}, ..., x_{k-1}] \\ 
\tag{1}
\end{aligned}
$$

$$\begin{array}{rl}{x}_1& =w_1\ast x_0\\ x_2& =w_2\ast [x_0,x_1]\\ ⋮& \\ x_k& =w_k\ast [x_0,x_1,...,x_{k-1}]\end{array}\text{\hspace{1em}(1)}$$

$$
\begin{aligned}
w_{1}^{,} &= f(w_{1}, \mathcal{g}_{0}) \\
w_{2}^{,} &= f(w_{2}, \mathcal{g}_{0}, \mathcal{g}_{1}) \\ 
w_{3}^{,} &= f(w_{3}, \mathcal{g}_{0}, \mathcal{g}_{1}, \mathcal{g}_{2}) \\ 
\vdots \\
w_{k}^{,} &= f(w_{k}, \mathcal{g}_{0}, \mathcal{g}_{1}, \mathcal{g}_{2}, ..., \mathcal{g}_{k-1}) \\ 
\tag{2}
\end{aligned}
$$

$$\begin{array}{rl}{w}_1^{,}& =f(w_1,g_0)\\ w_2^{,}& =f(w_2,g_0,g_1)\\ w_3^{,}& =f(w_3,g_0,g_1,g_2)\\ ⋮& \\ w_k^{,}& =f(w_k,g_0,g_1,g_2,...,g_{k-1})\end{array}\text{\hspace{1em}(2)}$$

$$
\begin{aligned}
x_{k} &= w_{k} * [x_{0}^{,,}, x_{1}, ..., x_{k-1}] \\
x_{T} &= w_{T} * [x_{0}^{,,}, x_{1}, ..., x_{k}] \\ 
x_{U} &= w_{U} * [x_{0}^{,}, x_{T}] \\ 
\tag{3}
\end{aligned}
$$

$$\begin{array}{rl}{x}_k& =w_k\ast [x_0^{,,},x_1,...,x_{k-1}]\\ x_T& =w_T\ast [x_0^{,,},x_1,...,x_k]\\ x_U& =w_U\ast [x_0^{,},x_T]\end{array}\text{\hspace{1em}(3)}$$

$$
\begin{aligned}
w_{k}^{,} &= f(w_{k}, \mathcal{g}_{0}^{,,}, \mathcal{g}_{1}, \mathcal{g}_{2}, ..., \mathcal{g}_{k-1}) \\
w_{T}^{,} &= f(w_{T}, \mathcal{g}_{0}^{,,}, \mathcal{g}_{1}, \mathcal{g}_{2}, ..., \mathcal{g}_{k}) \\ 
w_{U}^{,} &= f(w_{U}, \mathcal{g}_{0}^{,}, \mathcal{g}_{T}) \\ 
\tag{4}
\end{aligned}
$$

$$\begin{array}{rl}{w}_k^{,}& =f(w_k,g_0^{,,},g_1,g_2,...,g_{k-1})\\ w_T^{,}& =f(w_T,g_0^{,,},g_1,g_2,...,g_k)\\ w_U^{,}& =f(w_U,g_0^{,},g_T)\end{array}\text{\hspace{1em}(4)}$$

References

[1] Yongqiang Cheng, https://yongqiang.blog.csdn.net/