常用的范数求导

最新推荐文章于 2024-07-31 14:47:45 发布

机器学习的小学生

最新推荐文章于 2024-07-31 14:47:45 发布

阅读量7.1w

点赞数 52

分类专栏：数学知识

本文链接：https://blog.csdn.net/raby_gyl/article/details/53635459

版权

数学知识专栏收录该内容

3 篇文章 1 订阅

订阅专栏

矢量范数的偏导数

L1范数不可微。但是存在次梯度，即是次微分的。
L1范数的次梯度如下：
$\partial \partial x | | x | | 1 = s i g n (x)$ $\begin{equation} \begin{aligned} \frac{\partial }{\partial \mathbf x }||\mathbf x||_1=sign(\mathbf x) \end{aligned} \end{equation}$
其中 $sign(\mathbf x)$ 表示如下：
$s i g n (x) = ⎧ ⎩ ⎨ + 1 - 1 [- 1, 1] x i > 0 x i < 0 x i = 0$ $\begin{equation} sign(\mathbf x) = \begin{cases} +1 & x_i>0\\ -1& x_i <0\\ [-1,1]&x_i =0 \end{cases} \end{equation}$

在实验中，我们经常很碰到，一个函数表达式中含有多个带有绝对值表达式，我们为了去掉绝对值号，进行化简，经常需要假设函数绝对值中的表达式满足>0或者<0来消去绝对值。但是当变量很多时，很难划分这样的空间。例如上面的 $L_1$ 就是一个例子:

| | x | | 1 = | x 1 | + | x 2 | + . . . . + | x n |

$||\mathbf x||_1=|x_1|+|x_2|+....+|x_n|$
对于一维的情况：

| x | = {x - x x \geq 0 x \leq 0

$\begin{equation} |x|= \begin{cases} x & x \geq 0\\ -x& x \leq 0 \end{cases} \end{equation}$
但是对于高维的情况，我们很难写书上面明确的展开式。但是，在数值计算中，除了所谓的符号运。都是在知道明确的“值”的情况下，来进行求解的。
因此，知道了具体的值，我们很容易确定这个值的梯度。例如，对于3维的情况。如果值为

x0=[3,−2,5]T $x0 = [3, -2, 5]^T$ ，我们很容易知道，其函数值展开的表达式为：

| | x | | 1 = x 1 - x 2 + x 3

$||\mathbf x||_1 = x_1 - x_2 + x_3$
故其梯度为

[1,−1,1]T $[1,-1,1]^T$ ，即

sign(x) $sign(\mathbf x)$ 。
2. L2 范数：

\partial \partial x | | x - a | | 2 = x - a | | x - a | | 2

$\begin{equation} \frac{\partial }{\partial \mathbf x }||\mathbf x- \mathbf a||_2 =\frac{\mathbf x -\mathbf a}{||\mathbf x -\mathbf a||_2} \end{equation}$

\partial \partial x x - a | | x - a | | 2 = I | | x - a | | 2 - ( x - a ) ( x - a ) T | | x - a | | 3 2

$\begin{equation} \frac{\partial }{\partial \mathbf x }\frac{\mathbf x -\mathbf a }{||\mathbf x -\mathbf a||_2} = \frac{\mathbf I}{||\mathbf x-\mathbf a||_2 } -\frac{(\mathbf x -\mathbf a)(\mathbf x- \mathbf a)^T}{||\mathbf x-\mathbf a||_2^3} \end{equation}$

\partial | | x | | 2 2 \partial x = \partial | | x T x | | 2 \partial x = 2 x

$\begin{equation} \frac{\partial ||\mathbf x||_2^2}{\partial \mathbf x} = \frac{\partial ||\mathbf x^T \mathbf x||_2}{\partial \mathbf x} = 2 \mathbf x \end{equation}$

例如：求解下面函数的偏导数：

f (W) = 1 2 \sum i, j \in S 1 γ i, j | | w T i X - w T j X | | 22

$\begin{equation} f(\mathbf W) = \frac{1}{2} \sum_{i,j\in S_1} \gamma_{i,j} ||\mathbf w_i^T \mathbf X- \mathbf w_j^T \mathbf X||_2^2 \end{equation}$
其中

W $\mathbf W$ 是矩阵，大小

D×L $D \times L$ ，

X $\mathbf X$ 是矩阵,大小为

D×N $D \times N$ ，其中

D $D$ 是特征向量的维度，

L $L$ 是任务的数量，

N $N$ 是样本的数量。则矢量*矩阵，即

wTiX $\mathbf w_i^T \mathbf X$ 是一个矢量，矢量和矢量也是矢量，故这是要求解矢量L2范数的偏导数。

\partial f ( W ) \partial w i = = = \sum i, j \in S 1 γ i, j (w T i X - w T j X) * \partial ( w T i X - w T j X ) \partial w i \sum i, j \in S 1 γ i, j (w T i X - w T j X) * X T \sum i, j \in S 1 γ i, j (w T i - w T j) * (X X T)

$\begin{equation} \begin{aligned} \frac{\partial f(\mathbf W)}{\partial \mathbf w_i} =& \sum_{i,j\in S_1}\gamma_{i,j}(\mathbf w_i^T \mathbf X -\mathbf w_j^T\mathbf X)*\frac{\partial (\mathbf w_i^T \mathbf X -\mathbf w_j^T\mathbf X)}{\partial \mathbf w_i} \\ =& \sum_{i,j\in S_1}\gamma_{i,j}(\mathbf w_i^T \mathbf X -\mathbf w_j^T\mathbf X)*\mathbf X^T \\ =& \sum_{i,j\in S_1}\gamma_{i,j}(\mathbf w_i^T -\mathbf w_j^T)*(\mathbf X\mathbf X^T) \end{aligned} \end{equation}$

注意这里得到的是行向量的形式，因此还需要对其进行转置 。
对 $\mathbf w_j$ 求偏导数：

\partial f ( W ) \partial w j = = = \sum i, j \in S 1 γ i, j (w T j X - w T i X) * \partial ( w T j X - w T i X ) \partial w j \sum i, j \in S 1 γ i, j (w T j X - w T i X) * X T \sum i, j \in S 1 γ i, j (w T j - w T i) * (X X T)

$\begin{equation} \begin{aligned} \frac{\partial f(\mathbf W)}{\partial \mathbf w_j} =& \sum_{i,j\in S_1}\gamma_{i,j}(\mathbf w_j^T \mathbf X -\mathbf w_i^T\mathbf X)*\frac{\partial (\mathbf w_j^T \mathbf X -\mathbf w_i^T\mathbf X)}{\partial \mathbf w_j} \\ =& \sum_{i,j\in S_1}\gamma_{i,j}(\mathbf w_j^T \mathbf X -\mathbf w_i^T\mathbf X)*\mathbf X^T \\ =& \sum_{i,j\in S_1}\gamma_{i,j}(\mathbf w_j^T -\mathbf w_i^T)*(\mathbf X\mathbf X^T) \end{aligned} \end{equation}$

Matlab对低维数据进行验证:

% verify
syms x11 x12 x21 x22 x31 x32 wi1 wi2 wi3 wj1 wj2 wj3 real;% define symbols as real value

%method 1
X = [x11 x12;x21 x22;x31 x32]; % X 3*2 ,two sampels ,feature dimension 3
wi = [wi1 wi2 wi3]';
wj = [wj1 wj2 wj3]';

fw = 1/2*norm(wi'*X-wj'*X,2).^2;

grad_wi1 = diff(fw,wi1); %\partial wi1
grad_wi2 = diff(fw,wi2); %\partial wi2
grad_wi3 = diff(fw,wi3); %\partial wi3
grad_wi0 =[grad_wi1;grad_wi2;grad_wi3];

% method 2
grad_wi = (wi'*X-wj'*X)*X';
grad_wi = grad_wi';

clc;
disp('method 1:');
disp(grad_wi0);
disp('method 2:');
disp(grad_wi);

disp('%%%%%%%%%%%%%%%%%%%%%% \partial wj %%%%%%%%%%%%%%%%%%');

% method 1
grad_wj1 = diff(fw,wj1);
grad_wj2 = diff(fw,wj2);
grad_wj3 = diff(fw,wj3);
grad_wj0 = [grad_wj1;grad_wj2;grad_wj3];
% method2

grad_wj = (wj'*X- wi'*X)*X';
grad_wj = grad_wj';

disp('method 1:');
disp(grad_wj0);
disp('method 2:');
disp(grad_wj);

结果：

method 1:
 x11*abs(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31)*sign(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) + x12*abs(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)*sign(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)
 x21*abs(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31)*sign(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) + x22*abs(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)*sign(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)
 x31*abs(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31)*sign(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) + x32*abs(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)*sign(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)

method 2:
 x11*(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) + x12*(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)
 x21*(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) + x22*(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)
 x31*(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) + x32*(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)

%%%%%%%%%%%%%%%%%%%%%% \partial wj %%%%%%%%%%%%%%%%%%
method 1:
 - x11*abs(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31)*sign(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) - x12*abs(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)*sign(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)
 - x21*abs(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31)*sign(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) - x22*abs(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)*sign(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)
 - x31*abs(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31)*sign(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) - x32*abs(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)*sign(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)

method 2:
 - x11*(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) - x12*(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)
 - x21*(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) - x22*(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)
 - x31*(wi1*x11 + wi2*x21 + wi3*x31 - wj1*x11 - wj2*x21 - wj3*x31) - x32*(wi1*x12 + wi2*x22 + wi3*x32 - wj1*x12 - wj2*x22 - wj3*x32)