转载自AI算法工程师
一、基本知识
-
本书中所有的向量都是列向量的形式:
x ⃗ = ( x 1 , x 2 , ⋯   , x n ) T = [ x 1 x 2 ⋮ x n ] \mathbf{\vec x}=(x_1,x_2,\cdots,x_n)^T=\begin{bmatrix}x_1\\x_2\\ \vdots \\x_n\end{bmatrix} x=(x1,x2,⋯,xn)T=⎣⎢⎢⎢⎡x1x2⋮xn⎦⎥⎥⎥⎤
本书中所有的矩阵 X ∈ R m × n \mathbf X\in \mathbb R^{m\times n} X∈Rm×n 都表示为:
X = [ x 1 , 1 x 1 , 2 ⋯ x 1 , n x 2 , 1 x 2 , 2 ⋯ x 2 , n ⋮ ⋮ ⋱ ⋮ x m , 1 x m , 2 ⋯ x m , n ] \mathbf X = \begin{bmatrix} x_{1,1}&x_{1,2}&\cdots&x_{1,n}\\ x_{2,1}&x_{2,2}&\cdots&x_{2,n}\\ \vdots&\vdots&\ddots&\vdots\\ x_{m,1}&x_{m,2}&\cdots&x_{m,n}\\ \end{bmatrix} X=⎣⎢⎢⎢⎡x1,1x2,1⋮xm,1x1,2x2,2⋮xm,2⋯⋯⋱⋯x1,nx2,n⋮xm,n⎦⎥⎥⎥⎤
简写为: ( x i , j ) m × n (x_{i,j})_{m\times n} (xi,j)m×n或者 [ x i , j ] m × n [x_{i,j}]_{m\times n} [xi,j]m×n。
-
矩阵的
F
范数:设矩阵 A = ( a i , j ) m × n \mathbf A=(a_{i,j})_{m\times n} A=(ai,j)m×n,则其F
范数为: ∣ ∣ A ∣ ∣ F = ∑ i , j a i , j 2 ||\mathbf A||_F=\sqrt{\sum_{i,j}a_{i,j}^{2}} ∣∣A∣∣F=∑i,jai,j2。它是向量的 L 2 L_2 L2 范数的推广。
-
矩阵的迹:设矩阵 A = ( a i , j ) m × n \mathbf A=(a_{i,j})_{m\times n} A=(ai,j)m×n,则 A \mathbf A A的迹为: t r ( A ) = ∑ i a i , i tr(\mathbf A)=\sum_{i}a_{i,i} tr(A)=∑iai,i。
迹的性质有:
-
A
\mathbf A
A 的
F
范数等于 A A T \mathbf A\mathbf A^T AAT 的迹的平方根:$||\mathbf A||_F=\sqrt{tr(\mathbf A \mathbf A^{T})} $。 - A \mathbf A A 的迹等于$\mathbf A^T 的 迹 : 的迹: 的迹:tr(\mathbf A)=tr(\mathbf A^{T})$ 。
- 交换律:假设 A ∈ R m × n , B ∈ R n × m , \mathbf A\in \mathbb R^{m\times n},\mathbf B\in \mathbb R^{n\times m}, A∈Rm×n,B∈Rn×m,则有: t r ( A B ) = t r ( B A ) tr(\mathbf A\mathbf B)=tr(\mathbf B\mathbf A) tr(AB)=tr(BA) 。
- 结合律: t r ( A B C ) = t r ( C A B ) = t r ( B C A ) 。 tr(\mathbf A\mathbf B\mathbf C)=tr(\mathbf C\mathbf A\mathbf B)=tr(\mathbf B\mathbf C\mathbf A) 。 tr(ABC)=tr(CAB)=tr(BCA)。
-
A
\mathbf A
A 的
二、向量操作
-
一组向量 v ⃗ 1 , v ⃗ 2 , ⋯   , v ⃗ n \mathbf{\vec v}_1,\mathbf{\vec v}_2,\cdots,\mathbf{\vec v}_n v1,v2,⋯,vn是线性相关的:指存在一组不全为零的实数 a 1 , a 2 , ⋯   , a n a_1,a_2,\cdots,a_n a1,a2,⋯,an使得: ∑ i = 1 n a i v ⃗ i = 0 ⃗ \sum_{i=1}^{n}a_i\mathbf{\vec v}_i=\mathbf{\vec 0} ∑i=1naivi=0。
一组向量 v ⃗ 1 , v ⃗ 2 , ⋯   , v ⃗ n \mathbf{\vec v}_1,\mathbf{\vec v}_2,\cdots,\mathbf{\vec v}_n v1,v2,⋯,vn是线性无关的,当且仅当 a i = 0 , i = 1 , 2 , ⋯   , n a_i=0,i=1,2,\cdots,n ai=0,i=1,2,⋯,n时,才有: ∑ i = 1 n a i v ⃗ i = 0 ⃗ \sum_{i=1}^{n}a_i\mathbf{\vec v}_i=\mathbf{\vec 0} ∑i=1naivi=0。
-
一个向量空间所包含的最大线性无关向量的数目,称作该向量空间的维数。
-
三维向量的点积: u ⃗ ⋅ v ⃗ = u x v x + u y v y + u z v z = ∣ u ⃗ ∣ ∣ v ⃗ ∣ cos ( u ⃗ , v ⃗ ) \mathbf{\vec u}\cdot\mathbf{\vec v} =u _xv_x+u_yv_y+u_zv_z = |\mathbf{\vec u}| | \mathbf{\vec v}| \cos(\mathbf{\vec u},\mathbf{\vec v}) u⋅v=uxvx+uyvy+uzvz=∣u∣∣v∣cos(u,v)。
-
三维向量的叉积:
w ⃗ = u ⃗ × v ⃗ = [ i ⃗ j ⃗ k ⃗ u x u y u z v x v y v z ] \mathbf{\vec w}=\mathbf{\vec u}\times \mathbf{\vec v}=\begin{bmatrix}\mathbf{\vec i}& \mathbf{\vec j}&\mathbf{\vec k}\\ u_x&u_y&u_z\\ v_x&v_y&v_z\\ \end{bmatrix} w=u×v=⎣⎡iuxvxjuyvykuzvz⎦⎤
其中 i ⃗ , j ⃗ , k ⃗ \mathbf{\vec i}, \mathbf{\vec j},\mathbf{\vec k} i,j,k 分别为$ x,y,z $轴的单位向量。
u ⃗ = u x i ⃗ + u y j ⃗ + u z k ⃗ , v ⃗ = v x i ⃗ + v y j ⃗ + v z k ⃗ \mathbf{\vec u}=u_x\mathbf{\vec i}+u_y\mathbf{\vec j}+u_z\mathbf{\vec k},\quad \mathbf{\vec v}=v_x\mathbf{\vec i}+v_y\mathbf{\vec j}+v_z\mathbf{\vec k} u=uxi+uyj+uzk,v=vxi+vyj+vzk
- u ⃗ \mathbf{\vec u} u和 v ⃗ \mathbf{\vec v} v的叉积垂直于 u ⃗ , v ⃗ \mathbf{\vec u},\mathbf{\vec v} u,v构成的平面,其方向符合右手规则。
- 叉积的模等于 u ⃗ , v ⃗ \mathbf{\vec u},\mathbf{\vec v} u,v构成的平行四边形的面积
- u ⃗ × v ⃗ = − v ⃗ × u ⃗ \mathbf{\vec u}\times \mathbf{\vec v}=-\mathbf{\vec v}\times \mathbf{\vec u} u×v=−v×u
- u ⃗ × ( v ⃗ × w ⃗ ) = ( u ⃗ ⋅ w ⃗ ) v ⃗ − ( u ⃗ ⋅ v ⃗ ) w ⃗ \mathbf{\vec u}\times( \mathbf{\vec v} \times \mathbf{\vec w})=(\mathbf{\vec u}\cdot \mathbf{\vec w})\mathbf{\vec v}-(\mathbf{\vec u}\cdot \mathbf{\vec v})\mathbf{\vec w} u×(v×w)=(u⋅w)v−(u⋅v)w
-
三维向量的混合积:
[ u ⃗    v ⃗    w ⃗ ] = ( u ⃗ × v ⃗ ) ⋅ w ⃗ = u ⃗ ⋅ ( v ⃗ × w ⃗ ) = ∣ u x u y u z v x v y v z w x w y w z ∣ = ∣ u x v x w x u y v y w y u z v z w z ∣ [\mathbf{\vec u} \;\mathbf{\vec v} \;\mathbf{\vec w}]=(\mathbf{\vec u}\times \mathbf{\vec v})\cdot \mathbf{\vec w}= \mathbf{\vec u}\cdot (\mathbf{\vec v} \times \mathbf{\vec w})\\ =\begin{vmatrix} u_x&u_y&u_z\\ v_x&v_y&v_z\\ w_x&w_y&w_z \end{vmatrix} =\begin{vmatrix} u_x&v_x&w_x\\ u_y&v_y&w_y\\ u_z&v_z&w_z\end{vmatrix} [uvw]=(u×v)⋅w=u⋅(v×w)=∣∣∣∣∣∣uxvxwxuyvywyuzvzwz∣∣∣∣∣∣=∣∣∣∣∣∣uxuyuzvxvyvzwxwywz∣∣∣∣∣∣
其物理意义为: 以 u ⃗ , v ⃗ , w ⃗ 以 \mathbf{\vec u} ,\mathbf{\vec v} ,\mathbf{\vec w} 以u,v,w为三个棱边所围成的平行六面体的体积。 当 u ⃗ , v ⃗ , w ⃗ \mathbf{\vec u} ,\mathbf{\vec v} ,\mathbf{\vec w} u,v,w构成右手系时,该平行六面体的体积为正号。
-
两个向量的并矢:给定两个向量 x ⃗ = ( x 1 , x 2 , ⋯   , x n ) T , y ⃗ = ( y 1 , y 2 , ⋯   , y m ) T \mathbf {\vec x}=(x_1,x_2,\cdots,x_n)^{T}, \mathbf {\vec y}= (y_1,y_2,\cdots,y_m)^{T} x=(x1,x2,⋯,xn)T,y=(y1,y2,⋯,ym)T,则向量的并矢记作:
x ⃗ y ⃗ = [ x 1 y 1 x 1 y 2 ⋯ x 1 y m x 2 y 1 x 2 y 2 ⋯ x 2 y m ⋮ ⋮ ⋱ ⋮ x n y 1 x n y 2 ⋯ x n y m ] \mathbf {\vec x}\mathbf {\vec y} =\begin{bmatrix}x_1y_1&x_1y_2&\cdots&x_1y_m\\ x_2y_1&x_2y_2&\cdots&x_2y_m\\ \vdots&\vdots&\ddots&\vdots\\ x_ny_1&x_ny_2&\cdots&x_ny_m\\ \end{bmatrix} xy=⎣⎢⎢⎢⎡x1y1x2y1⋮xny1x1y2x2y2⋮xny2⋯⋯⋱⋯x1ymx2ym⋮xnym⎦⎥⎥⎥⎤
也记作 x ⃗ ⊗ y ⃗ \mathbf {\vec x}\otimes\mathbf {\vec y} x⊗y 或者 x ⃗ y ⃗ T \mathbf {\vec x} \mathbf {\vec y}^{T} xyT。
三、矩阵运算
-
给定两个矩阵 A = ( a i , j ) ∈ R m × n , B = ( b i , j ) ∈ R m × n \mathbf A=(a_{i,j}) \in \mathbb R^{m\times n},\mathbf B=(b_{i,j}) \in \mathbb R^{m\times n} A=(ai,j)∈Rm×n,B=(bi,j)∈Rm×n ,定义:
-
阿达马积
Hadamard product
(又称作逐元素积):A ∘ B = [ a 1 , 1 b 1 , 1 a 1 , 2 b 1 , 2 ⋯ a 1 , n b 1 , n a 2 , 1 b 2 , 1 a 2 , 2 b 2 , 2 ⋯ a 2 , n b 2 , n ⋮ ⋮ ⋱ ⋮ a m , 1 b m , 1 a m , 2 b m , 2 ⋯ a m , n b m , n ] \mathbf A \circ \mathbf B =\begin{bmatrix} a_{1,1}b_{1,1}&a_{1,2}b_{1,2}&\cdots&a_{1,n}b_{1,n}\\ a_{2,1}b_{2,1}&a_{2,2}b_{2,2}&\cdots&a_{2,n}b_{2,n}\\ \vdots&\vdots&\ddots&\vdots\\ a_{m,1}b_{m,1}&a_{m,2}b_{m,2}&\cdots&a_{m,n}b_{m,n}\end{bmatrix} A∘B=⎣⎢⎢⎢⎡a1,1b1,1a2,1b2,1⋮am,1bm,1a1,2b1,2a2,2b2,2⋮am,2bm,2⋯⋯⋱⋯a1,nb1,na2,nb2,n⋮am,nbm,n⎦⎥⎥⎥⎤
-
克罗内积
Kronnecker product
:A ⊗ B = [ a 1 , 1 B a 1 , 2 B ⋯ a 1 , n B a 2 , 1 B a 2 , 2 B ⋯ a 2 , n B ⋮ ⋮ ⋱ ⋮ a m , 1 B a m , 2 B ⋯ a m , n B ] \mathbf A \otimes \mathbf B =\begin{bmatrix}a_{1,1}\mathbf B&a_{1,2}\mathbf B&\cdots&a_{1,n}\mathbf B\\ a_{2,1}\mathbf B&a_{2,2}\mathbf B&\cdots&a_{2,n}\mathbf B\\ \vdots&\vdots&\ddots&\vdots\\ a_{m,1}\mathbf B&a_{m,2}\mathbf B&\cdots&a_{m,n}\mathbf B \end{bmatrix} A⊗B=⎣⎢⎢⎢⎡a1,1Ba2,1B⋮am,1Ba1,2Ba2,2B⋮am,2B⋯⋯⋱⋯a1,nBa2,nB⋮am,nB⎦⎥⎥⎥⎤
-
-
设 x ⃗ , a ⃗ , b ⃗ , c ⃗ \mathbf {\vec x},\mathbf {\vec a},\mathbf {\vec b},\mathbf {\vec c} x,a,b,c为 n 阶向量, A , B , C , X \mathbf A,\mathbf B,\mathbf C,\mathbf X A,B,C,X为 n 阶方阵,则有:
∂ ( a ⃗ T x ⃗ ) ∂ x ⃗ = ∂ ( x ⃗ T a ⃗ ) ∂ x ⃗ = a ⃗ \frac{\partial(\mathbf {\vec a}^{T}\mathbf {\vec x}) }{\partial \mathbf {\vec x} }=\frac{\partial(\mathbf {\vec x}^{T}\mathbf {\vec a}) }{\partial \mathbf {\vec x} } =\mathbf {\vec a} ∂x∂(aTx)=∂x∂(xTa)=a
∂ ( a ⃗ T X b ⃗ ) ∂ X = a ⃗ b ⃗ T = a ⃗ ⊗ b ⃗ ∈ R n × n \frac{\partial(\mathbf {\vec a}^{T}\mathbf X\mathbf {\vec b}) }{\partial \mathbf X }=\mathbf {\vec a}\mathbf {\vec b}^{T}=\mathbf {\vec a}\otimes\mathbf {\vec b}\in \mathbb R^{n\times n} ∂X∂(aTXb)=abT=a⊗b∈Rn×n
∂ ( a ⃗ T X T b ⃗ ) ∂ X = b ⃗ a ⃗ T = b ⃗ ⊗ a ⃗ ∈ R n × n \frac{\partial(\mathbf {\vec a}^{T}\mathbf X^{T}\mathbf {\vec b}) }{\partial \mathbf X }=\mathbf {\vec b}\mathbf {\vec a}^{T}=\mathbf {\vec b}\otimes\mathbf {\vec a}\in \mathbb R^{n\times n} ∂X∂(aTXTb)=baT=b⊗a∈Rn×n
∂ ( a ⃗ T X a ⃗ ) ∂ X = ∂ ( a ⃗ T X T a ⃗ ) ∂ X = a ⃗ ⊗ a ⃗ \frac{\partial(\mathbf {\vec a}^{T}\mathbf X\mathbf {\vec a}) }{\partial \mathbf X }=\frac{\partial(\mathbf {\vec a}^{T}\mathbf X^{T}\mathbf {\vec a}) }{\partial \mathbf X }=\mathbf {\vec a}\otimes\mathbf {\vec a} ∂X∂(aTXa)=∂X∂(aTXTa)=a⊗a
∂ ( a ⃗ T X T X b ⃗ ) ∂ X = X ( a ⃗ ⊗ b ⃗ + b ⃗ ⊗ a ⃗ ) \frac{\partial(\mathbf {\vec a}^{T}\mathbf X^{T}\mathbf X\mathbf {\vec b}) }{\partial \mathbf X }=\mathbf X(\mathbf {\vec a}\otimes\mathbf {\vec b}+\mathbf {\vec b}\otimes\mathbf {\vec a}) ∂X∂(aTXTXb)=X(a⊗b+b⊗a)
∂ [ ( A x ⃗ + a ⃗ ) T C ( B x ⃗ + b ⃗ ) ] ∂ x ⃗ = A T C ( B x ⃗ + b ⃗ ) + B T C ( A x ⃗ + a ⃗ ) \frac{\partial[(\mathbf A\mathbf {\vec x}+\mathbf {\vec a})^{T}\mathbf C(\mathbf B\mathbf {\vec x}+\mathbf {\vec b})]}{\partial \mathbf {\vec x}}=\mathbf A^{T}\mathbf C(\mathbf B\mathbf {\vec x}+\mathbf {\vec b})+\mathbf B^{T}\mathbf C(\mathbf A\mathbf {\vec x}+\mathbf {\vec a}) ∂x∂[(Ax+a)TC(Bx+b)]=ATC(Bx+b)+BTC(Ax+a)
∂ ( x ⃗ T A x ⃗ ) ∂ x ⃗ = ( A + A T ) x ⃗ \frac{\partial (\mathbf {\vec x}^{T}\mathbf A \mathbf {\vec x})}{\partial \mathbf {\vec x}}=(\mathbf A+\mathbf A^{T})\mathbf {\vec x} ∂x∂(xTAx)=(A+AT)x
∂ [ ( X b ⃗ + c ⃗ ) T A ( X b ⃗ + c ⃗ ) ] ∂ X = ( A + A T ) ( X b ⃗ + c ⃗ ) b ⃗ T \frac{\partial[(\mathbf X\mathbf {\vec b}+\mathbf {\vec c})^{T}\mathbf A(\mathbf X\mathbf {\vec b}+\mathbf {\vec c})]}{\partial \mathbf X}=(\mathbf A+\mathbf A^{T})(\mathbf X\mathbf {\vec b}+\mathbf {\vec c})\mathbf {\vec b}^{T} ∂X∂[(Xb+c)TA(Xb+c)]=(A+AT)(Xb+c)bT
∂ ( b ⃗ T X T A X c ⃗ ) ∂ X = A T X b ⃗ c ⃗ T + A X c ⃗ b ⃗ T \frac{\partial (\mathbf {\vec b}^{T}\mathbf X^{T}\mathbf A \mathbf X\mathbf {\vec c})}{\partial \mathbf X}=\mathbf A^{T}\mathbf X\mathbf {\vec b}\mathbf {\vec c}^{T}+\mathbf A\mathbf X\mathbf {\vec c}\mathbf {\vec b}^{T} ∂X∂(bTXTAXc)=ATXbcT+AXcbT
-
如果 f 是一元函数,则:
-
其逐元向量函数为: f ( x ⃗ ) = ( f ( x 1 ) , f ( x 2 ) , ⋯   , f ( x n ) ) T f(\mathbf{\vec x}) =(f(x_1),f(x_2),\cdots,f(x_n))^{T} f(x)=(f(x1),f(x2),⋯,f(xn))T 。
-
其逐矩阵函数为:
f ( X ) = [ f ( x 1 , 1 ) f ( x 1 , 2 ) ⋯ f ( x 1 , n ) f ( x 2 , 1 ) f ( x 2 , 2 ) ⋯ f ( x 2 , n ) ⋮ ⋮ ⋱ ⋮ f ( x m , 1 ) f ( x m , 2 ) ⋯ f ( x m , n ) ] f(\mathbf X)=\begin{bmatrix} f(x_{1,1})&f(x_{1,2})&\cdots&f(x_{1,n})\\ f(x_{2,1})&f(x_{2,2})&\cdots&f(x_{2,n})\\ \vdots&\vdots&\ddots&\vdots\\ f(x_{m,1})&f(x_{m,2})&\cdots&f(x_{m,n})\\ \end{bmatrix} f(X)=⎣⎢⎢⎢⎡f(x1,1)f(x2,1)⋮f(xm,1)f(x1,2)f(x2,2)⋮f(xm,2)⋯⋯⋱⋯f(x1,n)f(x2,n)⋮f(xm,n)⎦⎥⎥⎥⎤
-
其逐元导数分别为:
f ′ ( x ⃗ ) = ( f ′ ( x 1 ) , f ′ ( x 2 ) , ⋯   , f ′ ( x n ) ) T f ′ ( X ) = [ f ′ ( x 1 , 1 ) f ′ ( x 1 , 2 ) ⋯ f ′ ( x 1 , n ) f ′ ( x 2 , 1 ) f ′ ( x 2 , 2 ) ⋯ f ′ ( x 2 , n ) ⋮ ⋮ ⋱ ⋮ f ′ ( x m , 1 ) f ′ ( x m , 2 ) ⋯ f ′ ( x m , n ) ] f^{\prime}(\mathbf{\vec x}) =(f^{\prime}(x1),f^{\prime}(x2),\cdots,f^{\prime}(x_n))^{T}\\ f^{\prime}(\mathbf X)=\begin{bmatrix} f^{\prime}(x_{1,1})&f^{\prime}(x_{1,2})&\cdots&f^{\prime}(x_{1,n})\\ f^{\prime}(x_{2,1})&f^{\prime}(x_{2,2})&\cdots&f^{\prime}(x_{2,n})\\ \vdots&\vdots&\ddots&\vdots\\ f^{\prime}(x_{m,1})&f^{\prime}(x_{m,2})&\cdots&f^{\prime}(x_{m,n})\\ \end{bmatrix} f′(x)=(f′(x1),f′(x2),⋯,f′(xn))Tf′(X)=⎣⎢⎢⎢⎡f′(x1,1)f′(x2,1)⋮f′(xm,1)f′(x1,2)f′(x2,2)⋮f′(xm,2)⋯⋯⋱⋯f′(x1,n)f′(x2,n)⋮f′(xm,n)⎦⎥⎥⎥⎤
-
-
各种类型的偏导数:
-
标量对标量的偏导数: ∂ u ∂ v \frac{\partial u}{\partial v} ∂v∂u。
-
标量对向量(n 维向量)的偏导数 : ∂ u ∂ v ⃗ = ( ∂ u ∂ v 1 , ∂ u ∂ v 2 , ⋯   , ∂ u ∂ v n ) T \frac{\partial u}{\partial \mathbf {\vec v}}=(\frac{\partial u}{\partial v_1},\frac{\partial u}{\partial v_2},\cdots,\frac{\partial u}{\partial v_n})^{T} ∂v∂u=(∂v1∂u,∂v2∂u,⋯,∂vn∂u)T 。
-
标量对矩阵( m × n m\times n m×n阶矩阵)的偏导数:
∂ u ∂ V = [ ∂ u ∂ V 1 , 1 ∂ u ∂ V 1 , 2 ⋯ ∂ u ∂ V 1 , n ∂ u ∂ V 2 , 1 ∂ u ∂ V 2 , 2 ⋯ ∂ u ∂ V 2 , n ⋮ ⋮ ⋱ ⋮ ∂ u ∂ V m , 1 ∂ u ∂ V m , 2 ⋯ ∂ u ∂ V m , n ] \frac{\partial u}{\partial \mathbf V}=\begin{bmatrix} \frac{\partial u}{\partial V_{1,1}}&\frac{\partial u}{\partial V_{1,2}}&\cdots&\frac{\partial u}{\partial V_{1,n}}\\ \frac{\partial u}{\partial V_{2,1}}&\frac{\partial u}{\partial V_{2,2}}&\cdots&\frac{\partial u}{\partial V_{2,n}}\\ \vdots&\vdots&\ddots&\vdots\\ \frac{\partial u}{\partial V_{m,1}}&\frac{\partial u}{\partial V_{m,2}}&\cdots&\frac{\partial u}{\partial V_{m,n}} \end{bmatrix} ∂V∂u=⎣⎢⎢⎢⎢⎡∂V1,1∂u∂V2,1∂u⋮∂Vm,1∂u∂V1,2∂u∂V2,2∂u⋮∂Vm,2∂u⋯⋯⋱⋯∂V1,n∂u∂V2,n∂u⋮∂Vm,n∂u⎦⎥⎥⎥⎥⎤
-
向量(m 维向量)对标量的偏导数: ∂ u ⃗ ∂ v = ( ∂ u 1 ∂ v , ∂ u 2 ∂ v , ⋯   , ∂ u m ∂ v ) T \frac{\partial \mathbf {\vec u}}{\partial v}=(\frac{\partial u_1}{\partial v},\frac{\partial u_2}{\partial v},\cdots,\frac{\partial u_m}{\partial v})^{T} ∂v∂u=(∂v∂u1,∂v∂u2,⋯,∂v∂um)T。
-
向量(m 维向量)对向量 (n 维向量) 的偏导数(雅可比矩阵,行优先)
∂ u ⃗ ∂ v ⃗ = [ ∂ u 1 ∂ v 1 ∂ u 1 ∂ v 2 ⋯ ∂ u 1 ∂ v n ∂ u 2 ∂ v 1 ∂ u 2 ∂ v 2 ⋯ ∂ u 2 ∂ v n ⋮ ⋮ ⋱ ⋮ ∂ u m ∂ v 1 ∂ u m ∂ v 2 ⋯ ∂ u m ∂ v n ] \frac{\partial \mathbf {\vec u}}{\partial \mathbf {\vec v}}=\begin{bmatrix} \frac{\partial u_1}{\partial v_1}&\frac{\partial u_1}{\partial v_2}&\cdots&\frac{\partial u_1}{\partial v_n}\\ \frac{\partial u_2}{\partial v_1}&\frac{\partial u_2}{\partial v_2}&\cdots&\frac{\partial u_2}{\partial v_n}\\ \vdots&\vdots&\ddots&\vdots\\ \frac{\partial u_m}{\partial v_1}&\frac{\partial u_m}{\partial v_2}&\cdots&\frac{\partial u_m}{\partial v_n} \end{bmatrix} ∂v∂u=⎣⎢⎢⎢⎡∂v1∂u1∂v1∂u2⋮∂v1∂um∂v2∂u1∂v2∂u2⋮∂v2∂um⋯⋯⋱⋯∂vn∂u1∂vn∂u2⋮∂vn∂um⎦⎥⎥⎥⎤
如果为列优先,则为上面矩阵的转置。
-
矩阵( m × n m\times n m×n阶矩阵)对标量的偏导数
∂ U ∂ v = [ ∂ U 1 , 1 ∂ v ∂ U 1 , 2 ∂ v ⋯ ∂ U 1 , n ∂ v ∂ U 2 , 1 ∂ v ∂ U 2 , 2 ∂ v ⋯ ∂ U 2 , n ∂ v ⋮ ⋮ ⋱ ⋮ ∂ U m , 1 ∂ v ∂ U m , 2 ∂ v ⋯ ∂ U m , n ∂ v ] \frac{\partial \mathbf U}{\partial v}=\begin{bmatrix} \frac{\partial U_{1,1}}{\partial v}&\frac{\partial U_{1,2}}{\partial v}&\cdots&\frac{\partial U_{1,n}}{\partial v}\\ \frac{\partial U_{2,1}}{\partial v}&\frac{\partial U_{2,2}}{\partial v}&\cdots&\frac{\partial U_{2,n}}{\partial v}\\ \vdots&\vdots&\ddots&\vdots\\ \frac{\partial U_{m,1}}{\partial v}&\frac{\partial U_{m,2}}{\partial v}&\cdots&\frac{\partial U_{m,n}}{\partial v} \end{bmatrix} ∂v∂U=⎣⎢⎢⎢⎢⎡∂v∂U1,1∂v∂U2,1⋮∂v∂Um,1∂v∂U1,2∂v∂U2,2⋮∂v∂Um,2⋯⋯⋱⋯∂v∂U1,n∂v∂U2,n⋮∂v∂Um,n⎦⎥⎥⎥⎥⎤
-
-
对于矩阵的迹,有下列偏导数成立:
∂ [ t r ( f ( X ) ) ] ∂ X = ( f ′ ( X ) ) T \frac{\partial [tr(f(\mathbf X))]}{\partial \mathbf X }=(f^{\prime}(\mathbf X))^{T} ∂X∂[tr(f(X))]=(f′(X))T
∂ [ t r ( A X B ) ] ∂ X = A T B T \frac{\partial [tr(\mathbf A\mathbf X\mathbf B)]}{\partial \mathbf X }=\mathbf A^{T}\mathbf B^{T} ∂X∂[tr(AXB)]=ATBT
∂ [ t r ( A X T B ) ] ∂ X = B A \frac{\partial [tr(\mathbf A\mathbf X^{T}\mathbf B)]}{\partial \mathbf X }=\mathbf B\mathbf A ∂X∂[tr(AXTB)]=BA
∂ [ t r ( A ⊗ X ) ] ∂ X = t r ( A ) I \frac{\partial [tr(\mathbf A\otimes\mathbf X )]}{\partial \mathbf X }=tr(\mathbf A)\mathbf I ∂X∂[tr(A⊗X)]=tr(A)I
∂ [ t r ( A X B X ) ] ∂ X = A T X T B T + B T X A T \frac{\partial [tr(\mathbf A\mathbf X \mathbf B\mathbf X)]}{\partial \mathbf X }=\mathbf A^{T}\mathbf X^{T}\mathbf B^{T}+\mathbf B^{T}\mathbf X \mathbf A^{T} ∂X∂[tr(AXBX)]=ATXTBT+BTXAT
∂ [ t r ( X T B X C ) ] ∂ X = B X C + B T X C T \frac{\partial [tr(\mathbf X^{T} \mathbf B\mathbf X \mathbf C)]}{\partial \mathbf X }=\mathbf B\mathbf X \mathbf C +\mathbf B^{T}\mathbf X \mathbf C^{T} ∂X∂[tr(XTBXC)]=BXC+BTXCT
∂ [ t r ( C T X T B X C ) ] ∂ X = ( B T + B ) X C C T \frac{\partial [tr(\mathbf C^{T}\mathbf X^{T} \mathbf B\mathbf X \mathbf C)]}{\partial \mathbf X }=(\mathbf B^{T}+\mathbf B)\mathbf X \mathbf C \mathbf C^{T} ∂X∂[tr(CTXTBXC)]=(BT+B)XCCT
∂ [ t r ( A X B X T C ) ] ∂ X = A T C T X B T + C A X B \frac{\partial [tr(\mathbf A\mathbf X \mathbf B\mathbf X^{T} \mathbf C)]}{\partial \mathbf X }= \mathbf A^{T}\mathbf C^{T}\mathbf X\mathbf B^{T}+\mathbf C \mathbf A \mathbf X \mathbf B ∂X∂[tr(AXBXTC)]=ATCTXBT+CAXB
∂ [ t r ( ( A X B + C ) ( A X B + C ) ) ] ∂ X = 2 A T ( A X B + C ) B T \frac{\partial [tr((\mathbf A\mathbf X\mathbf B+\mathbf C)(\mathbf A\mathbf X\mathbf B+\mathbf C))]}{\partial \mathbf X }= 2\mathbf A ^{T}(\mathbf A\mathbf X\mathbf B+\mathbf C)\mathbf B^{T} ∂X∂[tr((AXB+C)(AXB+C))]=2AT(AXB+C)BT
-
假设 U = f ( X ) \mathbf U= f(\mathbf X) U=f(X)是关于 X \mathbf X X 的矩阵值函数 ( f : R m × n → R m × n ) (f:\mathbb R^{m\times n}\rightarrow \mathbb R^{m\times n}) (f:Rm×n→Rm×n),且 g ( U ) g(\mathbf U) g(U)是关于 U \mathbf U U的实值
函数 ( g : R m × n → R ) (g:\mathbb R^{m\times n} \rightarrow \mathbb R) (g:Rm×n→R),则下面链式法则成立:
∂
g
(
U
)
∂
X
=
(
∂
g
(
U
)
∂
x
i
,
j
)
m
×
n
=
[
∂
g
(
U
)
∂
x
1
,
1
∂
g
(
U
)
∂
x
1
,
2
⋯
∂
g
(
U
)
∂
x
1
,
n
∂
g
(
U
)
∂
x
2
,
1
∂
g
(
U
)
∂
x
2
,
2
⋯
∂
g
(
U
)
∂
x
2
,
n
⋮
⋮
⋱
⋮
∂
g
(
U
)
∂
x
m
,
1
∂
g
(
U
)
∂
x
m
,
2
⋯
∂
g
(
U
)
∂
x
m
,
n
]
\frac{\partial g(\mathbf U)}{\partial \mathbf X}= \left(\frac{\partial g(\mathbf U)}{\partial x_{i,j}}\right)_{m\times n}=\begin{bmatrix} \frac{\partial g(\mathbf U)}{\partial x_{1,1}}&\frac{\partial g(\mathbf U)}{\partial x_{1,2}}&\cdots&\frac{\partial g(\mathbf U)}{\partial x_{1,n}}\\ \frac{\partial g(\mathbf U)}{\partial x_{2,1}}&\frac{\partial g(\mathbf U)}{\partial x_{2,2}}&\cdots&\frac{\partial g(\mathbf U)}{\partial x_{2,n}}\\ \vdots&\vdots&\ddots&\vdots\\ \frac{\partial g(\mathbf U)}{\partial x_{m,1}}&\frac{\partial g(\mathbf U)}{\partial x_{m,2}}&\cdots&\frac{\partial g(\mathbf U)}{\partial x_{m,n}}\end{bmatrix}
∂X∂g(U)=(∂xi,j∂g(U))m×n=⎣⎢⎢⎢⎢⎢⎡∂x1,1∂g(U)∂x2,1∂g(U)⋮∂xm,1∂g(U)∂x1,2∂g(U)∂x2,2∂g(U)⋮∂xm,2∂g(U)⋯⋯⋱⋯∂x1,n∂g(U)∂x2,n∂g(U)⋮∂xm,n∂g(U)⎦⎥⎥⎥⎥⎥⎤
=
(
∑
k
∑
l
∂
g
(
U
)
∂
u
k
,
l
∂
u
k
,
l
∂
x
i
,
j
)
m
×
n
=
(
t
r
[
(
∂
g
(
U
)
∂
U
)
T
∂
U
∂
x
i
,
j
]
)
m
×
n
\left(\sum_{k}\sum_{l}\frac{\partial g(\mathbf U)}{\partial u_{k,l}}\frac{\partial u_{k,l}}{\partial x_{i,j}}\right)_{m\times n}=\left(tr\left[\left(\frac{\partial g(\mathbf U)}{\partial \mathbf U}\right)^{T}\frac{\partial \mathbf U}{\partial x_{i,j}}\right]\right)_{m\times n}
(∑k∑l∂uk,l∂g(U)∂xi,j∂uk,l)m×n=(tr[(∂U∂g(U))T∂xi,j∂U])m×n
四、特殊函数
- 这里给出机器学习中用到的一些特殊函数。
4.1 sigmoid 函数
-
sigmoid
函数:σ ( x ) = 1 1 + exp ( − x ) \sigma(x)=\frac{1}{1+\exp(-x)} σ(x)=1+exp(−x)1
- 该函数可以用于生成二项分布的 ϕ \phi ϕ参数。
- 当 x 很大或者很小时,该函数处于饱和状态。此时函数的曲线非常平坦,并且自变量的一个较大的变化只能带来函数值的一个微小的变化,即:导数很小。
4.2 softplus 函数
-
softplus
函数: ζ ( x ) = log ( 1 + exp ( x ) ) \zeta(x)=\log(1+\exp(x)) ζ(x)=log(1+exp(x))。- 该函数可以生成正态分布的 σ 2 \sigma^{2} σ2参数。
- 它之所以称作
softplus
,因为它是下面函数的一个光滑逼近: x + = max ( 0 , x ) x^{+}=\max(0,x) x+=max(0,x)。
-
如果定义两个函数:
x + = max ( 0 , x ) x − = max ( 0 , − x ) x^{+}=\max(0,x)\\ x^{-}=\max(0,-x) x+=max(0,x)x−=max(0,−x)
则它们分布获取了 y=x 的正部分和负部分。
根据定义有: x = x + − x − x=x^{+}-x^{-} x=x+−x− 。而 ζ ( x ) \zeta(x) ζ(x)逼近的是 x + x^{+} x+, ζ ( − x ) \zeta(-x) ζ(−x)逼近的是 x − x^{-} x−,于是有:
ζ ( x ) − ζ ( − x ) = x \zeta(x)-\zeta(-x)=x ζ(x)−ζ(−x)=x
-
sigmoid
和softplus
函数的性质:σ ( x ) = exp ( x ) exp ( x ) + exp ( 0 ) d d x σ ( x ) = σ ( x ) ( 1 − σ ( x ) ) 1 − σ ( x ) = σ ( − x ) log σ ( x ) = − ζ ( − x ) d d x ζ ( x ) = σ ( x ) ∀ x ∈ ( 0 , 1 ) , σ − 1 ( x ) = log ( x 1 − x ) ∀ x > 0 , ζ − 1 ( x ) = log ( exp ( x ) − 1 ) ζ ( x ) = ∫ − ∞ x σ ( y ) d y ζ ( x ) − ζ ( − x ) = x \sigma(x)=\frac{\exp(x)}{\exp(x)+\exp(0)} \\ \frac {d}{dx}\sigma(x)=\sigma(x)(1-\sigma(x)) \\ 1-\sigma(x)=\sigma(-x) \\ \log\sigma(x)=-\zeta(-x) \\ \frac{d}{dx}\zeta(x)=\sigma(x) \\ \forall x\in(0,1),\sigma^{-1}(x)=\log(\frac{x}{1-x}) \\ \forall x \gt 0,\zeta^{-1}(x)=\log(\exp(x)-1) \\ \zeta(x)=\int_{-\infty}^{x}\sigma(y)dy \\ \zeta(x)-\zeta(-x)=x σ(x)=exp(x)+exp(0)exp(x)dxdσ(x)=σ(x)(1−σ(x))1−σ(x)=σ(−x)logσ(x)=−ζ(−x)dxdζ(x)=σ(x)∀x∈(0,1),σ−1(x)=log(1−xx)∀x>0,ζ−1(x)=log(exp(x)−1)ζ(x)=∫−∞xσ(y)dyζ(x)−ζ(−x)=x
其中 f − 1 ( ⋅ ) f^{-1}(\cdot) f−1(⋅)为反函数。
σ − 1 ( x ) \sigma^{-1}(x) σ−1(x)也称作
logit
函数。
4.3 伽马函数
-
伽马函数定义为:
Γ ( x ) = ∫ 0 + ∞ t x − 1 e − t d t , x ∈ R o r . Γ ( z ) = ∫ 0 + ∞ t z − 1 e − t d t , z ∈ Z \Gamma(x)=\int_0^{+\infty} t^{x-1}e^{-t}dt\quad,x\in \mathbb R\\ or. \quad\Gamma(z)=\int_0^{+\infty} t^{z-1}e^{-t}dt\quad,z\in \mathbb Z Γ(x)=∫0+∞tx−1e−tdt,x∈Ror.Γ(z)=∫0+∞tz−1e−tdt,z∈Z
性质为:
-
对于正整数 n 有: Γ ( n ) = ( n − 1 ) ! \Gamma(n)=(n-1)! Γ(n)=(n−1)!。
-
Γ ( x + 1 ) = x Γ ( x ) \Gamma(x+1)=x\Gamma(x) Γ(x+1)=xΓ(x),因此伽马函数是阶乘在实数域上的扩展。
-
与贝塔函数的关系:
B ( m , n ) = Γ ( m ) Γ ( n ) Γ ( m + n ) B(m,n)=\frac{\Gamma(m)\Gamma(n)}{\Gamma(m+n)} B(m,n)=Γ(m+n)Γ(m)Γ(n)
-
对于 x ∈ ( 0 , 1 ) x \in (0,1) x∈(0,1) 有:
Γ ( 1 − x ) Γ ( x ) = π sin π x \Gamma(1-x)\Gamma(x)=\frac{\pi}{\sin\pi x} Γ(1−x)Γ(x)=sinπxπ
则可以推导出重要公式: Γ ( 1 2 ) = π \Gamma(\frac 12)=\sqrt\pi Γ(21)=π。
-
对于 x > 0 x\gt 0 x>0,伽马函数是严格凹函数。
-
-
当 x 足够大时,可以用
Stirling
公式来计算Gamma
函数值: Γ ( x ) ∼ 2 π e − x x x + 1 / 2 \Gamma(x) \sim \sqrt{2\pi} e^{-x}x^{x+1/2} Γ(x)∼2πe−xxx+1/2。
4.4 贝塔函数
-
对于任意实数$ m,n \gt 0 $,定义贝塔函数:
B ( m , n ) = ∫ 0 1 x m − 1 ( 1 − x ) n − 1 d x B(m,n)=\int_0^1 x^{m-1}(1-x)^{n-1} dx B(m,n)=∫01xm−1(1−x)n−1dx
其它形式的定义:
B ( m , n ) = 2 ∫ 0 π 2 sin 2 m − 1 ( x ) cos 2 n − 1 ( x ) d x B ( m , n ) = ∫ 0 + ∞ x m − 1 ( 1 + x ) m + n d x B ( m , n ) = ∫ 0 1 x m − 1 + x n − 1 ( 1 + x ) m + n d x B(m,n)=2\int_0^{\frac \pi 2}\sin ^{2m-1}(x) \cos ^{2n-1}(x) dx\\ B(m,n) = \int_0^{+\infty}\frac{x^{m-1}}{(1+x)^{m+n}} dx\\ B(m,n)=\int_0^1\frac{x^{m-1}+x^{n-1}}{(1+x)^{m+n}}dx B(m,n)=2∫02πsin2m−1(x)cos2n−1(x)dxB(m,n)=∫0+∞(1+x)m+nxm−1dxB(m,n)=∫01(1+x)m+nxm−1+xn−1dx
-
性质:
-
连续性:贝塔函数在定义域$ m\gt0,n\gt0$ 内连续。
-
对称性:B(m,n)=B(n,m) 。
-
递个公式:
B ( m , n ) = n − 1 m + n − 1 B ( m , n − 1 ) , m > 0 , n > 1 B ( m , n ) = m − 1 m + n − 1 B ( m − 1 , n ) , m > 1 , n > 0 B ( m , n ) = ( m − 1 ) ( n − 1 ) ( m + n − 1 ) ( m + n − 2 ) B ( m − 1 , n − 1 ) , m > 1 , n > 1 B(m,n) = \frac{n-1}{m+n-1}B(m,n-1),\quad m\gt0,n\gt1\\ B(m,n) = \frac{m-1}{m+n-1}B(m-1,n),\quad m\gt1,n\gt0\\ B(m,n) = \frac{(m-1)(n-1)}{(m+n-1)(m+n-2)}B(m-1,n-1),\quad m\gt1,n\gt1 B(m,n)=m+n−1n−1B(m,n−1),m>0,n>1B(m,n)=m+n−1m−1B(m−1,n),m>1,n>0B(m,n)=(m+n−1)(m+n−2)(m−1)(n−1)B(m−1,n−1),m>1,n>1
-
当 m,n 较大时,有近似公式:
B ( m , n ) = ( 2 π ) m m − 1 / 2 n n − 1 / 2 ( m + n ) m + n − 1 / 2 B(m,n)=\frac{\sqrt{(2\pi)m^{m-1/2}n^{n-1/2}}}{(m+n)^{m+n-1/2}} B(m,n)=(m+n)m+n−1/2(2π)mm−1/2nn−1/2
-
与伽马函数关系:
-
对于任意正实数 m,n ,有:
B ( m , n ) = Γ ( m ) Γ ( n ) Γ ( m + n ) B(m,n)=\frac{\Gamma(m)\Gamma(n)}{\Gamma(m+n)} B(m,n)=Γ(m+n)Γ(m)Γ(n)
-
B ( m , 1 − m ) = Γ ( m ) Γ ( 1 − m ) 。 B(m,1-m)=\Gamma(m)\Gamma(1-m) 。 B(m,1−m)=Γ(m)Γ(1−m)。
-
-