Softmax回归求导公式推导

原来推导过，但是好像没有推导清楚，看了别人博文，感觉其推导有误，重新推导一下（以MNIST手写数字图片数据尺寸为例），练练脑力！

1. Softmax回归计算公式

1）线性加权

$$O_j=\sum_{i=1}^{784}W_{ij}*X_i \qquad j=1,...,10$$
2）softmax计算概率
$$y_j=\frac{exp(O_j)}{\sum_{k=1}^{10}exp(O_k)}\qquad j=1,...,10$$
3）交叉熵
$$L(y)=-\sum_{j=1}^{10}\hat{y}_j \log y_j$$
4）偏导数
$$\frac{\partial L}{\partial W_{ij}}=\frac{\partial L}{\partial O_{j}}\frac{\partial O_j}{\partial W_{ij}}=\frac{\partial L}{\partial O_{j}}X_i$$

$$\frac{\partial L}{\partial O_{j}}=-\sum_{k=1}^{10}\frac{\partial L}{\partial y_{k}}\frac{\partial y_k}{\partial O_{j}}=-\sum_{k=1}^{10}\hat{y}_k \frac{1}{y_k}\frac{\partial y_k}{\partial O_{j}}$$

$$\frac{\partial y_k}{\partial O_{j}}=\frac{exp(O_k)\frac{\partial O_k}{\partial O_j}\sum_{m=1}^{10}exp(O_m)-exp(O_k)exp(O_j)}{\lbrack\sum_{m=1}^{10}exp(O_m)\rbrack^2}$$

$$\frac{\partial y_k}{\partial O_{j}}=y_k\frac{\frac{\partial O_k}{\partial O_j}\sum_{m=1}^{10}exp(O_m)-exp(O_j)}{\sum_{m=1}^{10}exp(O_m)}=y_k\frac{\partial O_k}{\partial O_j}-y_ky_j$$

$$\frac{\partial L}{\partial O_{j}}=\sum_{k=1}^{10}\hat{y}_k\frac{\partial O_k}{\partial O_j}-\lgroup\sum_{k=1}^{10}\hat{y}_k\rgroup y_j=\hat{y}_j-y_j$$

偏导数的矩阵表达形式为：

$$\frac{\partial L}{\partial W}=(\hat{Y}-Y)^TX$$
其中：
$\hat{Y}$的维度为 $1\times 10$
$X$的维度为$1\times 784$
$\frac{\partial L}{\partial W}$的维度为 $10\times 784$