回归和分类求导练习

前置知识

矩阵求导
https://blog.csdn.net/qq_39942341/article/details/128739604?spm=1001.2014.3001.5502
（看微分那部分就够了）

回归

设$\mathbf{X}\in {\mathbb{R}}^{B\times m},{\mathbf{W}}_1\in {\mathbb{R}}^{n\times m},1\in {\mathbb{R}}^{n\times 1},{\mathbf{b}}_1\in {\mathbb{R}}^{1\times n},{\mathbf{Y}}_1\in {\mathbb{R}}^{B\times n}$
${\mathbf{W}}_2\in {\mathbb{R}}^{p\times n},{\mathbf{b}}_1\in {\mathbb{R}}^{1\times p},{\mathbf{Y}}_2\in {\mathbb{R}}^{B\times p}$
$\sigma \left(\cdot \right)$是激活函数，例如sigmoid

$$\begin{gathered} {\mathbf{Y}}_1=\mathbf{X}{\mathbf{W}}_1^T+1{\mathbf{b}}_1 \\ {\mathbf{A}}_1=\sigma \left({\mathbf{Y}}_1\right) \\ {\mathbf{Y}}_2={\mathbf{A}}_1{\mathbf{W}}_2^T+1{\mathbf{b}}_2 \\ {\mathbf{A}}_2=\sigma \left({\mathbf{Y}}_2\right) \\ l=\frac{1}{2}mse\left(\mathbf{A},{\mathbf{A}}_2\right)=\frac{1}{2}\Vert \mathbf{A}-{\mathbf{A}}_2{\Vert }_F^2 \end{gathered}$$
$$\frac{\partial l}{\partial {\mathbf{A}}_2}={\mathbf{A}}_2-\mathbf{A}$$
$$\begin{array}{rl}\mathrm{d}\mathrm{l}& =tr\left({\frac{\partial l}{\partial {\mathbf{A}}_2}}^T\mathrm{d}{\mathbf{A}}_2\right)\\ & =tr\left({\frac{\partial l}{\partial {\mathbf{A}}_2}}^T\mathrm{d}\sigma \left({\mathbf{Y}}_2\right)\right)\\ & =tr\left({\frac{\partial l}{\partial {\mathbf{A}}_2}}^T{\sigma }^{\prime }\left({\mathbf{Y}}_2\right)\mathrm{d}{\mathbf{Y}}_2\right)\\ & =tr\left({\left(\frac{\partial l}{\partial {\mathbf{A}}_2}\odot {\sigma }^{\prime }\left({\mathbf{Y}}_2\right)\right)}^T\mathrm{d}{\mathbf{Y}}_2\right)\\ & =tr\left({\frac{\partial l}{\partial {\mathbf{Y}}_2}}^T\mathrm{d}{\mathbf{Y}}_2\right)\end{array}$$
因此
$$\frac{\partial l}{\partial {\mathbf{Y}}_2}=\frac{\partial l}{\partial {\mathbf{A}}_2}\odot {\sigma }^{\prime }\left({\mathbf{Y}}_2\right)$$

$$\begin{array}{rl}\mathrm{d}\mathrm{l}& =tr\left({\frac{\partial l}{\partial {\mathbf{Y}}_2}}^T\mathrm{d}{\mathbf{Y}}_2\right)\\ & =tr\left({\frac{\partial l}{\partial {\mathbf{Y}}_2}}^T\mathrm{d}\left({\mathbf{A}}_1{\mathbf{W}}_2^{\mathrm{T}}+1{\mathbf{b}}_2\right)\right)\\ & =tr\left({\frac{\partial l}{\partial {\mathbf{Y}}_2}}^T\left(\mathrm{d}{\mathbf{A}}_1\right){\mathbf{W}}_2^T\right)+tr\left({\frac{\partial l}{\partial {\mathbf{Y}}_2}}^T{\mathbf{A}}_1\left(\mathrm{d}{\mathbf{W}}_2^{\mathrm{T}}\right)\right)+tr\left({\frac{\partial l}{\partial {\mathbf{Y}}_2}}^{T}1\mathrm{d}\left(\mathrm{d}{\mathbf{b}}_2\right)\right)\\ & =tr\left({\mathbf{W}}_2^T{\frac{\partial l}{\partial {\mathbf{Y}}_2}}^T\left(\mathrm{d}{\mathbf{A}}_1\right)\right)+tr\left(\left(\mathrm{d}{\mathbf{W}}_2^{\mathrm{T}}\right){\frac{\partial l}{\partial {\mathbf{Y}}_2}}^T{\mathbf{A}}_1\right)+tr\left({\frac{\partial l}{\partial {\mathbf{Y}}_2}}^{T}1\mathrm{d}\left(\mathrm{d}{\mathbf{b}}_2\right)\right)\end{array}$$
因此
$$\begin{gathered} \frac{\partial l}{\partial {\mathbf{A}}_1}=\frac{\partial l}{\partial {\mathbf{Y}}_2}{\mathbf{W}}_2 \\ \frac{\partial l}{\partial {\mathbf{W}}_2}={\frac{\partial l}{\partial {\mathbf{Y}}_2}}^T{\mathbf{A}}_1 \\ \frac{\partial l}{\partial {\mathbf{b}}_2}=1^T\frac{\partial l}{\partial {\mathbf{Y}}_2} \end{gathered}$$
同理
$$\begin{gathered} \frac{\partial l}{\partial {\mathbf{Y}}_1}=\frac{\partial l}{\partial {\mathbf{A}}_1}\odot {\sigma }^{\prime }\left({\mathbf{Y}}_1\right) \\ \frac{\partial l}{\partial {\mathbf{W}}_2}={\frac{\partial l}{\partial {\mathbf{Y}}_1}}^T\mathbf{X} \\ \frac{\partial l}{\partial {\mathbf{b}}_1}=1^T\frac{\partial l}{\partial {\mathbf{Y}}_1} \end{gathered}$$
如果采用sigmoid，则${\sigma }^{\prime }\left(\mathbf{X}\right)=\sigma \left(\mathbf{X}\right)\left(1-\sigma \left(\mathbf{X}\right)\right)$
如果采用relu，则${\left[{\sigma }^{\prime }\left(\mathbf{X}\right)\right]}_{ij}=\{\begin{array}{l}1,X_{ij}>0\\ 0,otherwise\end{array}$

代码验证

#!/usr/bin/env python
# _*_ coding:utf-8 _*_
import torch
from torch import nn


def sigmoid_derivative(Y):
    return Y * (1 - Y)


def relu_derivative(Y):
    return torch.where(Y > 0, 1, 0)


if __name__ == '__main__':
    B, m, n, p = 3, 5, 4, 6
    linear1 = nn.Linear(m, n)
    active1 = nn.Sigmoid()
    derivative_1 = sigmoid_derivative
    linear2 = nn.Linear(n, p)
    active2 = nn.ReLU()
    derivative_2 = relu_derivative

    A = torch.randn(B, p)

    X = torch.randn(B, m, requires_grad=True)
    Y1 = linear1(X)
    A1 = active1(Y1)

    Y2 = linear2(A1)
    A2 = active2(Y2)

    # 1/2 mse(A2, A)
    l = torch.sum((A2 - A) ** 2) * 0.5
    l.backward()

    grad_A2 = A2 - A
    grad_Y2 = grad_A2 * derivative_2(A2)
    grad_W2 = torch.mm(grad_Y2.T, A1)
    grad_b2 = torch.mm(torch.ones(B, 1).T, grad_Y2)
    print(torch.allclose(grad_W2, linear2.weight.grad))
    print(torch.allclose(grad_b2, linear2.bias.grad))

    grad_A1 = torch.mm(grad_Y2, linear2.weight)
    grad_Y1 = grad_A1 * derivative_1(A1)
    grad_W1 = torch.mm(grad_Y1.T, X)
    grad_b1 = torch.mm(torch.ones(B, 1).T, grad_Y1)
    print(torch.allclose(grad_W1, linear1.weight.grad))
    print(torch.allclose(grad_b1, linear1.bias.grad))

分类

对于行向量$\mathbf{a}\in {\mathbb{R}}^{1\times n}$
$$softmax\left(\mathbf{a}\right)=\frac{e^{\mathbf{a}}}{e^{\mathbf{a}}{1}_n}$$
其中$1_n\in {\mathbb{R}}^n$,为全1向量
设$\mathbf{y}\in {\mathbb{R}}^{1\times n}$只有一个元素为1，其他元素为0
交叉熵
$$\begin{array}{rl}ce\left(\mathbf{a},\mathbf{y}\right)& =-\log \left(softmax\left(\mathbf{a}\right)\right){\mathbf{y}}^T\\ & =-\left(\mathbf{a}-\log \left(e^{\mathbf{a}}{1}_n\right)1_n^T\right){\mathbf{y}}^T\\ & =-\mathbf{a}{\mathbf{y}}^T+\log \left(e^{\mathbf{a}}{1}_n\right)\end{array}$$
求导
$$\begin{array}{rl}\mathrm{d}\mathrm{l}& =tr\left(-\left(\mathrm{d}\mathbf{a}\right){\mathbf{y}}^T+\frac{1}{e^{\mathbf{a}}{1}_n}\left(e^{\mathbf{a}}\odot \mathrm{d}\mathbf{a}\right)1_n\right)\\ & =tr\left(-\left(\mathrm{d}\mathbf{a}\right){\mathbf{y}}^T+\frac{1}{e^{\mathbf{a}}{1}_n}{\left(1_n^T\right)}^T\left(e^{\mathbf{a}}\odot \mathrm{d}\mathbf{a}\right)\right)\\ & =tr\left(-\left(\mathrm{d}\mathbf{a}\right){\mathbf{y}}^T+\frac{1}{e^{\mathbf{a}}{1}_n}{\left(1_n^T\odot e^{\mathbf{a}}\right)}^T\left(\mathrm{d}\mathbf{a}\right)\right)\\ & =tr\left(-\left(\mathrm{d}\mathbf{a}\right){\mathbf{y}}^T+\frac{1}{e^{\mathbf{a}}{1}_n}{\left(e^{\mathbf{a}}\right)}^T\left(\mathrm{d}\mathbf{a}\right)\right)\\ & =tr\left(-{\mathbf{y}}^T\left(\mathrm{d}\mathbf{a}\right)+{\left(softmax\left(\mathbf{a}\right)\right)}^T\left(\mathrm{d}\mathbf{a}\right)\right)\end{array}$$
于是
$$\frac{\partial l}{\partial \mathbf{a}}=softmax\left(\mathbf{a}\right)-\mathbf{y}$$

设$\mathbf{A}\in {\mathbb{R}}^{B\times n},\mathbf{Y}\in {\mathbb{R}}^{B\times n}$,
其中$\mathbf{Y}$每行只有一个元素为1，其他元素为0
设${\mathbf{a}}_i$表示$\mathbf{A}$第$i$行
设${\mathbf{y}}_i$表示$\mathbf{Y}$第$i$行
$softmax\left(\mathbf{A}\right)=\left(\begin{array}{c}softmax\left({\mathbf{a}}_1\right)\\ softmax\left({\mathbf{a}}_2\right)\\ ⋮\\ softmax\left({\mathbf{a}}_B\right)\end{array}\right)$
设$1_B\in {\mathbb{R}}^B$，为全1向量
$ce\left(\mathbf{A},\mathbf{Y}\right)={\sum }_{i=1}^{B}ce\left({\mathbf{a}}_i,{\mathbf{y}}_i\right)=1_B^T\log \left(e^{\mathbf{A}}{1}_n\right)-tr\left(\mathbf{A}{\mathbf{Y}}^T\right)$
求导得
$$\frac{\partial l}{\partial \mathbf{A}}=\left(\begin{array}{c}\frac{\partial l}{\partial {\mathbf{a}}_1}\\ \frac{\partial l}{\partial {\mathbf{a}}_2}\\ ⋮\\ \frac{\partial l}{\partial {\mathbf{a}}_B}\end{array}\right)=\left(\begin{array}{c}softmax\left({\mathbf{a}}_1\right)-{\mathbf{y}}_1\\ softmax\left({\mathbf{a}}_2\right)-{\mathbf{y}}_2\\ ⋮\\ softmax\left({\mathbf{a}}_B\right)-{\mathbf{y}}_B\end{array}\right)=softmax\left(\mathbf{A}\right)-\mathbf{Y}$$
验证：

#!/usr/bin/env python
# _*_ coding:utf-8 _*_
import torch
from torch import nn
import torch.nn.functional as F

if __name__ == '__main__':
    B, n = 3, 4
    ce = nn.CrossEntropyLoss(reduction='sum')
    target = torch.empty(B, dtype=torch.long).random_(n)
    target_one_hot = F.one_hot(target, num_classes=n)

    A = torch.randn(B, n, requires_grad=True)
    l = ce(A, target)
    l.backward()
    ones_B = torch.ones(B, 1)
    ones_n = torch.ones(n, 1)
    output = torch.mm(ones_B.T, torch.log(torch.mm(torch.exp(A), ones_n))) - (
        torch.mm(A, target_one_hot.T.float())).trace()
    print(torch.allclose(output, l))

    grad_A = F.softmax(A, dim=1) - target_one_hot
    print(torch.allclose(grad_A, A.grad))