c++ 实现linear和CNN神经网络（数学公式）

数学：
https://www.cnblogs.com/pinard/p/6422831.html
https://www.cnblogs.com/pinard/p/6494810.html
https://www.cnblogs.com/pinard/p/10750718.html
https://www.cnblogs.com/pinard/p/10773942.html
https://www.cnblogs.com/pinard/p/6494810.html
https://blog.csdn.net/qq_37951753/article/details/79672615
https://blog.csdn.net/evanxxxnnn/article/details/83552318
https://zhuanlan.zhihu.com/p/45310446
https://blog.csdn.net/qq_36342854/article/details/103863741
https://mp.weixin.qq.com/s/2xYgaeLlmmUfxiHCbCa8dQ
c++：
https://www.cnblogs.com/xuefeng00/p/11093425.html
http://www.cplusplus.com/reference/random/normal_distribution/
https://people.sc.fsu.edu/~jburkardt/cpp_src/truncated_normal/truncated_normal.html
https://www.cnblogs.com/jingshikongming/p/9037881.html
https://www.zhihu.com/question/63507542
https://blog.csdn.net/yahamatarge/article/details/89380164
https://www.cnblogs.com/jhmu0613/p/7750798.html
https://blog.csdn.net/qq_25175067/article/details/80266003
https://blog.csdn.net/lmb1612977696/article/details/80035487
http://blog.chinaunix.net/uid-20773165-id-1847733.html
https://baijiahao.baidu.com/s?id=1651645857687261494&wfr=spider&for=pc
https://blog.csdn.net/weixin_34007291/article/details/93528095

理论

多层感知机前向传播及导数

$$\begin{gathered} z^l=w^l{a}^{l-1}+b^l \\ {a}^l=\sigma (z^l) \\ {m}_l\times n=\sigma (m_l\times m_{l-1}\times m_{l-1}\times n+m_l\times 1) \\ (m_l,)=\sigma ((m_l,m_{l-1})\times (m_{l-1},)+(m_l,)) \end{gathered}$$

多层感知机反向传播

损失函数：
$$J(y,\hat{y})$$
导数传播
$$\begin{array}{rl}\frac{\partial J(y,\hat{y})}{\partial w^l}& =\frac{\partial J(y,\hat{y})}{\partial z^l}\frac{\partial z^l}{\partial w^l}\\ \frac{\partial J(y,\hat{y})}{\partial z^l}& 记为{\delta }^l\\ \frac{\partial J(y,\hat{y})}{\partial w^l}& ={\delta }^l\frac{\partial z^l}{\partial w^l}={\delta }^l{a^{l-1}}^T\\ \frac{\partial J(y,\hat{y})}{\partial z^L}& =▽J(y,\hat{y})\\ {\delta }^{L-1}=\frac{\partial J(y,\hat{y})}{\partial z^{L-1}}& =\frac{\partial J(y,\hat{y})}{\partial z^L}\frac{\partial z^L}{\partial a^{L-1}}\frac{\partial a^{L-1}}{\partial z^{L-1}}={w^L}^T{\delta }^L\odot {\sigma }^{{}^{\prime }}(z^{L-1})\\ \frac{\partial J(y,\hat{y})}{\partial b^l}& =\sum _m{\delta }_m^l\end{array}$$
所以每层 $l$ 维护
$$\frac{\partial z^l}{\partial w^l}=a^{l-1}$$

每层 $l$ 反向传播时返回
$$\frac{\partial J(y,\hat{y})}{\partial z^l}$$

CNN前向传播

$$\begin{gathered} 卷积: \\ {z}^l=a^{l-1}\ast W^l+b^l \\ {a}^l=\sigma (z^l) \\ 池化: \\ {a}^l=pooling(a^{l-1}) \end{gathered}$$

CNN反向传播

$$\begin{array}{rl}卷积:& \\ \frac{\partial J(y,\hat{y})}{\partial z^l}& =(\frac{\partial z^{l+1}}{\partial z^l}{)}^T\frac{\partial J(y,\hat{y})}{\partial z^{l+1}}\\ \frac{\partial J(y,\hat{y})}{\partial z^l}& ={\delta }^l\\ z^{l+1}& =a^l\ast W^{l+1}+b^{l+1}=\sigma (z^l)\ast W^{l+1}+b^{l+1}\\ \frac{\partial z^{l+1}}{\partial z^l}& =rot(W^{l+1})\odot {\sigma }^{{}^{\prime }}(z^l)\\ \frac{\partial J(y,\hat{y})}{\partial z^l}& ={\delta }^l={\delta }^{l+1}\ast rot(W^{l+1})\odot {\sigma }^{{}^{\prime }}(z^l)\\ \frac{\partial J(y,\hat{y})}{\partial w^l}& =a^{l-1}\ast {\delta }^l\\ \frac{\partial J(y,\hat{y})}{\partial b^l}& =\sum _{i,j}{\delta }_{i,j}^l\\ 池化：& \\ 将\frac{\partial J(y,\hat{y})}{\partial a^{l-1}}& 按照池化的权重回填成输入的矩阵\frac{\partial J(y,\hat{y})}{\partial a^l}\end{array}$$

每层 $l$ 维护：
$$雅克比矩阵：\frac{\partial J(y,\hat{y})}{\partial a^{l-1}}=(\frac{\partial z^l}{\partial a^{l-1}}{)}^T\frac{\partial J(y,\hat{y})}{\partial z^l}={\delta }^l\ast rot(W^l)$$

BatchNormal 前向传播

BatchNormal 反向传播

$$\begin{array}{rl}\frac{\partial J}{\partial \beta }& =\sum \frac{\partial J}{\partial y_i}\\ \frac{\partial J}{\partial \gamma }& =\sum \frac{\partial J}{\partial y_i}\widehat{x_i}\\ \frac{\partial C}{\partial x_i}& =\sum _k^m\frac{\partial C}{\partial {\hat{x}}_k}\frac{\partial {\hat{x}}_k}{\partial x_i}\\ & =\frac{\partial C}{\partial {\hat{x}}_i}\frac{1}{\sqrt{{\sigma }^2+ϵ}}+\sum _k^m\frac{\partial C}{\partial {\hat{x}}_k}\frac{\partial {\hat{x}}_k}{\partial {\sigma }^2}\frac{\partial {\sigma }^2}{\partial x_i}+\sum _k^m\frac{\partial C}{\partial {\hat{x}}_k}\frac{\partial {\hat{x}}_k}{\partial \mu }\frac{\partial \mu }{\partial x_i}\\ & =\frac{\partial C}{\partial {\hat{x}}_i}\frac{1}{\sqrt{{\sigma }^2+ϵ}}+\frac{\partial {\sigma }^2}{\partial x_i}\cdot \sum _k^m\frac{\partial C}{\partial {\hat{x}}_k}\frac{\partial {\hat{x}}_k}{\partial {\sigma }^2}+\frac{\partial \mu }{\partial x_i}\cdot \sum _k^m\frac{\partial C}{\partial {\hat{x}}_k}\frac{\partial {\hat{x}}_k}{\partial \mu }\\ \sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\frac{\partial {\hat{x}}_i}{\partial {\sigma }^2}& =\sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\left[-\frac{1}{2}\frac{x_i-\mu }{(\sqrt{{\sigma }^2+ϵ}{)}^3}\right]\\ & =-\frac{1}{2}\frac{1}{(\sqrt{{\sigma }^2+ϵ}{)}^3}\sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\left(x_i-\mu \right)\\ \frac{\partial {\sigma }^2}{\partial x_i}& =\frac{2}{m}\left(x_i-\mu \right)\\ \sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\frac{\partial {\hat{x}}_i}{\partial \mu }& =\frac{-1}{\sqrt{{\sigma }^2+ϵ}}\cdot \sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\\ \frac{\partial C}{\partial x_i}& =\frac{\partial C}{\partial {\hat{x}}_i}\frac{1}{\sqrt{{\sigma }^2+ϵ}}+\left[-\frac{1}{2}\frac{1}{(\sqrt{{\sigma }^2+ϵ}{)}^3}\sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\left(x_i-\mu \right)\right]\frac{2}{m}\left(x_i-\mu \right)\\ & +\frac{-1}{\sqrt{{\sigma }^2+ϵ}}\cdot \sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\frac{1}{m}\\ & =\frac{\partial C}{\partial {\hat{x}}_i}\frac{1}{\sqrt{{\sigma }^2+ϵ}}-\frac{1}{m}\frac{x_i-\mu }{(\sqrt{{\sigma }^2+ϵ}{)}^3}\sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\left(x_i-\mu \right)\\ & -\frac{1}{m}\frac{1}{\sqrt{{\sigma }^2+ϵ}}\cdot \sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\\ & =\frac{1}{m}\frac{1}{\sqrt{{\sigma }^2+ϵ}}\cdot \left\{m\frac{\partial C}{\partial {\hat{x}}_i}-\frac{x_i-\mu }{(\sqrt{{\sigma }^2+ϵ}{)}^2}\sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\left(x_i-\mu \right)-\sum _i^m\frac{\partial C}{\partial {\hat{x}}_i}\right\}\end{array}$$

softmax

原始公式：
$$y_i=\frac{x_i}{{\sum }_j{x}_j}$$
问题：
$e^n$ 当n稍微大点就爆精度了

解决：
$$\log \sum _i{e}^{x_i}=a+\log \sum _i{e}^{x_i-a}$$
$a$ 取 $x_i$ 中的最大值

初始化

truncated normal

产生截断正态分布随机数，取值范围为 [ mean - 2 * stddev, mean + 2 * stddev ]
只会循环产生随机数，将不满足的去掉
后来翻墙找到了一个库

实践

总目标是跑的快，可能牺牲代码可读性和安全性（瞎几把写）。

类设计

框架

1. Tensor 类, 维度，计算，名字，初始化
2. Layer类，输入，输出，参数，反向传播
   子类包括CNN、全连接、池化、batchnormal、激活和损失函数层
3. 网络类，一个计算图DAG，用拓扑排序挨个计算，每个节点都是layer，记录入度，包含损失函数类和train、test接口

细节

1. Tensor

元素

1. data[],一维
2. shape,vector
3. grad，null或者和data一样长
4. name，char数组，默认null

方法

1. 初始化
   1. 传入shape和初始化值
   2. 传入shape，random初始化
2. dot
   点乘 检查shape 遍历
3. mul
   矩阵乘法
   检查shape
4. add
   加法 检查shape
5. sub
   减法 检查shape
6. div
   除法 检查shape
7. print
   名字（如果有）维度一行
   数据一行
   梯度（如果有）一行
   可能重载流实现
8. setName、getName()
9. 重载运算符[]、()
   取出data和grad
10. reshape

2. Layer

元素

无

方法

1. forward
2. backward

子类1 linear

1. 神经元个数m，输入的维度
2. Tensor类，二维，参数W
3. Tensor类，参数bias
4. Tensor类，input

子类2 cnn

1. m,n,h,w，filter 个数，输入的channel，高宽
2. stride
3. padding
4. W 参数 （m,n,h,w）
5. bias 参数 （m,）

子类3 maxpooling

1. h,w,stride
2. 权重图，用于反向传播

子类4 softmax

子类5 relu

子类6 batchnormal

1. momentum (mean = momentum * mean + (1.0 - momentum) * nowbatchmean)
2. gamma, beta
3. mean，var
4. 形状以前一层是CNN还是linear来定

子类7 mse

子类8 cross validation loss

子类9 自定义loss

3. Net

元素

1. 图DAG，链式前向星还是vector再说
2. 每个节点是一个layer，但是要有输入输出buffer，防止分叉的情况重复计算

方法

1. init 建立好DAG，也就是不支持动态图
2. train 传入一个batch
3. test
4. ()重载运算符，前向传播
5. save
6. load