梯度下降法推导：多分类问题

数据集格式

在机器学习多分类问题里数据集格式一般如下:
第$i$个样本特征和标签写作：
$$\begin{gathered} x^i=(x_1^i,x_2^i,x_3^i,...,x_d^i{)}^T\in R^d \\ {y}^i=(y_1^i,y_2^i,y_3^i,...,y_m^i{)}^T\in R^m \end{gathered}$$
其中$d$代表输入的特征的维数，$m$代表输出类别的个数并对标签进行one-hot编码，
则完整的数据集可以写作：
$$\begin{gathered} X=[x^1,x^2,\dots ,x^n] \\ =\left[\begin{array}{cccc}{x}_1^1& x_1^2& \dots & x_1^n\\ x_2^1& x_2^2& \dots & x_2^n\\ ⋮& ⋮& \ddots & ⋮\\ x_d^1& x_d^2& \dots & x_d^n\end{array}\right]\in R^{d\ast n} \\ Y=[y^1,y^2,\dots ,y^n]\in R^{m\ast n} \end{gathered}$$

基于线性回归+sigmoid实现二分类的表达式

对于单个样本
$$\begin{gathered} o=Wx+b \\ =\left[\begin{array}{cccc}{w}_{11}& w_{12}& \dots & w_{1d}\\ w_{21}& w_{22}& \dots & w_{2d}\\ ⋮& ⋮& \ddots & ⋮\\ w_{m1}& w_{m2}& \dots & w_{md}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_d\end{array}\right]+\left[\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_m\end{array}\right] \\ =\left[\begin{array}{c}{w}_{11}\ast x_1+w_{12}\ast x_2+\dots +w_{1d}\ast x_d+b_1\\ w_{21}\ast x_1+w_{22}\ast x_2+\dots +w_{2d}\ast x_d+b_2\\ ⋮\\ w_{m1}\ast x_1+w_{m2}\ast x_2+\dots +w_{md}\ast x_d+b_m\end{array}\right] \end{gathered}$$
使用$softmax$函数实现输出为实现多分类，$softmax$函数表达式如下
给定
$$o=\left[\begin{array}{c}{o}_1\\ o_2\\ ⋮\\ o_m\end{array}\right]$$
$$\hat{y}=\left[\begin{array}{c}\widehat{y_1}\\ \widehat{y_2}\\ ⋮\\ \widehat{y_m}\end{array}\right]=softmax(o)=\left[\begin{array}{c}\frac{e^{o_1}}{{\sum }_{i=1}^m{e}^{o_i}}\\ \frac{e^{o_2}}{{\sum }_{i=1}^m{e}^{o_i}}\\ ⋮\\ \frac{e^{o_m}}{{\sum }_{i=1}^m{e}^{o_i}}\end{array}\right]$$
对于多分类问题，使用最小化负对数似然作为损失函数，其表达式为

$$L=-\log P(Y|X)=\sum _{i=1}^n-\log P(y^i|x^i)=\sum _{i=1}^{n}l(y_i,\widehat{y_i})$$
$l(y,\hat{y})$是针对于单个样本而定义的，具体写作
$$l(y,\hat{y})=-\sum _{j=1}^m{y}_j\log \widehat{y_j}$$
其中$y_j$为样本标签值，$\widehat{y_j}$为样本预测值，$m$为one-hot向量长度，代表分类种类数。

链式法则求导

链式表达式

求解$w_{jk}$和$b_j$的导数需要使用链式求导法则
求导公式如下：
$$\begin{gathered} \frac{\partial l}{\partial w_{jk}}=\frac{\partial l}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial o_j}\frac{\partial o_j}{\partial w_{jk}} \\ \frac{\partial l}{\partial b_j}=\frac{\partial l}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial o_j}\frac{\partial o_j}{\partial b_j} \end{gathered}$$

求解$\frac{\partial l}{\partial \widehat{o_j}}$

因为
$$\begin{gathered} l(y,\hat{y})=-\sum _{j=1}^m{y}_j\log \widehat{y_j} \\ \widehat{y_j}=\frac{e^{o_j}}{{\sum }_{i=1}^m{e}^{o_i}} \end{gathered}$$
则可得$l$关于$o_j$的表达式
$$\begin{gathered} l(y,\hat{y})=-\sum _{j=1}^m{y}_j\log \frac{e^{o_j}}{{\sum }_{i=1}^m{e}^{o_i}} \\ =\sum _{j=1}^m{y}_j\log \sum _{i=1}^m{e}^{o_i}-\sum _{j=1}^m{y}_j\log e^{o_j} \\ =\log \sum _{i=1}^m{e}^{o_i}-\sum _{j=1}^m{y}_j{o}_j \end{gathered}$$
因此
$$\frac{\partial l}{\partial o_j}=\frac{e^{o_j}}{{\sum }_{i=1}^m{e}^{o_i}}-y_j$$

求解$\frac{\partial o_j}{\partial w_{ij}}$

$o_j$关于$w_{jk}$的表达式可写作
$$o_j=w_{j1}\ast x_1+w_{j2}\ast x_2+\dots +w_{jd}\ast x_d+b_j$$
则
$$\frac{\partial o_j}{\partial w_{jk}}=x_k$$

求解$\frac{\partial o_j}{\partial b_j}$

$o_j$关于$b_j$的表达式为
$$o_j=w_{j1}\ast x_1+w_{j2}\ast x_2+\dots +w_{jd}\ast x_d+b_j$$
则可得
$$\frac{\partial o_j}{\partial b_j}=1$$

最终表达式

梯度表达式

$$\frac{\partial l}{\partial w_{jk}}=\frac{\partial l}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial o_j}\frac{\partial o_j}{\partial w_{jk}}=(\frac{e^{o_j}}{{\sum }_{i=1}^m{e}^{o_i}}-y_j)x_k$$
$$\frac{\partial l}{\partial b_j}=\frac{\partial l}{\partial \hat{y}}\frac{\partial \hat{y}}{\partial o_j}\frac{\partial o_j}{\partial b_j}=\frac{e^{o_j}}{{\sum }_{i=1}^m{e}^{o_i}}-y_j$$

梯度更新表达式

$w$更新表达式

因为
$$\frac{\partial l}{\partial w_{jk}}=(\frac{e^{o_j}}{{\sum }_{i=1}^m{e}^{o_i}}-y_j)x_k$$
则梯度更新表达式为
$$\begin{gathered} w_{jk}=w_{jk}-\eta \frac{\partial l}{\partial w_i} \\ =w_{jk}-\eta (\frac{e^{o_j}}{{\sum }_{i=1}^m{e}^{o_i}}-y_j)x_k \end{gathered}$$

$b$更新表达式

因为
$$\frac{\partial l}{\partial b_j}=\frac{e^{o_j}}{{\sum }_{i=1}^m{e}^{o_i}}-y_j$$
则梯度更新表达式为
$$\begin{gathered} b_j=b_j-\eta \frac{\partial l}{\partial b_j} \\ =b_j-\eta \frac{e^{o_j}}{{\sum }_{i=1}^m{e}^{o_i}}-y_j \end{gathered}$$

参考资料

Zhang, Aston and Lipton, Zachary C. and Li, Mu and Smola, Alexander J , DiveintoDeepLearning