23、机器学习数学基础：多元函数微积分

$y=f(x_1,x_2,...,x_n)，(x_1,x_2,...,x_n)\in R^n，x_i\in R$

点$(x_1,x_2,...,x_n)$与点$(z_1,z_2,...,z_n)$之间的距离为

$d=\sqrt{((x_1-z_1{)}^2+(x_2-z_2{)}^2+...+(x_n-z_n{)}^2}$

1、当函数通过不同方向趋近不同的极限时，该点的极限不存在

函数$f(x,y)=\frac{x^{2}y}{x^4+y^2}$,当沿着线性函数y=kx趋近于(0,0)这一点时，有$f(x,y)=\frac{x^3}{x^4+k^2{x}^2}$，当x趋近于0时，f(x,y)趋近于0。

$y=k{x}^2，f(x,y)=\frac{x^4}{x^4+k^2{x}^4}=\frac{k}{k^2+1}$不趋近于0。因为从不同方向趋近(0,0)时，有不同的值，所以f(x,y)在(0,0)极限不存在，也不连续。多维函数的极限是从整个范围来看，而不单单从某个方向。
$y=f(x_1,x_2,...,x_n)$极值，任意方向的变化都是变小，该点为极大值。

2、偏导数

对于$x_i$求导，$x_1,...,x_{i-1},x_{i+1},...x_n$固定，$\underset{x_i\to x_i^0}{\lim }\frac{f(x_1^0,x_2^0,...,x_{i-1}^0,x_i,x_{i+1}^0,x_n^0)-f(x_1^0,x_2^0,...,x_{i-1}^0,x_i^0,x_{i+1}^0,x_n^0)}{x_i-x_i^0}=\frac{\partial f(\overrightarrow{x_0})}{\partial x_i}$

函数在x、y、z上的偏导存在，函数不一定连续，例如：$f(x,y,z)=\{\begin{array}{l}0,xyz=0\\ 1,xyz\ne 0\end{array}$函数在(1,1,1)上沿方向轴(x,y,z)的偏导为0，在(0,0,0)上沿方向轴的偏导为0上沿方向轴(x,y,z)的偏导为0,(0,0,0)不连续，因为除了(0,0,0)和轴向、平面之外，其他地方都是1，不是对$\forall ϵ>0,\exists \delta >0$，当$|\vec{x}-\vec{0}|<\delta$有$|f(\vec{x})-f(\vec{0})|<ϵ,|f(\vec{x})-f(\vec{0})|=1$

3、方向导数

$\vec{v}=(x,y,z)$，$1=\sqrt{x^2+y^2+z^2},{\theta }_x$是$\vec{v}$与x轴的夹角，$\vec{v}=(cos({\theta }_x),cos({\theta }_y),cos({\theta }_z))$，$\underset{t\to 0}{\lim }\frac{f(t\vec{v}+\overrightarrow{x_0})-f(\overrightarrow{x_0})}{t}=$沿$\vec{v}$的方向导数，方向导数存在，偏导数也不一定存在，偏导数是沿x轴的方向，往x轴正向移动偏导数的变化与往负项移动偏导数的变化相反，正向和反向的方向导数是相反的。$f(x,y)=\sqrt{x^2+y^2}$沿任意方向走，方向导数是1，但是沿x轴正向走与沿负项走的都是1，所以在0点f(x,y)不存在x轴、y轴上的偏导数。

4、微分、梯度

$\Delta f(\overrightarrow{x_0})=f(\overrightarrow{x_0}+\Delta \vec{x})-f(\overrightarrow{x_0})={\sum }_{i=1}^n{A}_i\Delta x_i+o(|\Delta \vec{x}|)$存在的话，证明f(x)在$\overrightarrow{x_0}$的微分存在，该式为$\overrightarrow{x_0}$的全微分，如果在该点可微分，则在该点是连续的且偏导存在且为$A_i$，偏导存在且在每个方向上连续可推出函数可微。偏导是单个方向的变化，微分是每个方向变化的加和。

$\vec{v}=(cos{\theta }_1,cos{\theta }_2,...,cos{\theta }_n)$
如果函数可微，则方向函数$$\begin{gathered} \frac{\partial f(\overrightarrow{x_0})}{\partial \vec{v}}=\underset{t\to 0}{\lim }\frac{f(t\vec{v}+\overrightarrow{x_0})-f(\overrightarrow{x_0})}{t}=\underset{t\to 0}{\lim }\frac{\sum \frac{\partial f(x_0)}{\partial x_i}tcos{\theta }_i+o(|t|)+f(\overrightarrow{x_0})-f(\overrightarrow{x_0})}{t} \\ =\sum \frac{\partial f(x_0)}{\partial x_i}cos{\theta }_i=(\frac{\partial f(x_0)}{\partial x_1},\frac{\partial f(x_0)}{\partial x_2},...,\frac{\partial f(x_0)}{\partial x_n})(cos{\theta }_1,cos{\theta }_2,...,cos{\theta }_n)，(\frac{\partial f(x_0)}{\partial x_1},\frac{\partial f(x_0)}{\partial x_2},...,\frac{\partial f(x_0)}{\partial x_n}) \end{gathered}$$

为函数的梯度，决定了所有方向上方向导数的取值。向量函数$(f_1,f_2,...,f_n)$对每个分量$x_i$的偏导形成了雅克比矩阵，多元函数的梯度对应一元函数的导数。

4、链式法则

一元函数链式法则:

$y=f(u(x))=\frac{dy}{du}\frac{du}{dx}，y^{\prime }=f(u(x_0))=f^{\prime }(u(x_0))u^{\prime }(x_0)，\Delta y=\Delta f=f^{\prime }\Delta u=f^{\prime }{u}^{\prime }\Delta x，\frac{\Delta y}{\Delta x}=f^{\prime }(u(x_0))u^{\prime }$

多元函数链式法则: $f(u_1(\vec{x}),u_2(\vec{x}),...,u_n(\vec{x}))$，$u_1(x_1,x_2,...,x_n)$

$\Delta u_1=\frac{\partial u_1}{\partial x_1}\Delta x_1+\frac{\partial u_1}{\partial x_2}\Delta x_2...$

$\Delta y=\frac{\partial f}{\partial u_1}\Delta u_1+\frac{\partial f}{\partial u_2}\Delta u_2+...+\frac{\partial f}{\partial u_n}\Delta u_n=\frac{\partial f}{\partial u_1}\frac{\partial u_1}{\partial x_1}\Delta x_1+\frac{\partial f}{\partial u_2}\frac{\partial u_2}{\partial x_1}\Delta x_1...+\frac{\partial f}{\partial u_n}\frac{\partial u_n}{\partial x_1}\Delta x_1+()x_2...$

$\Delta y=A_1\Delta x_1+A_2\Delta x_2+...+A_n\Delta x_n$

$\frac{\partial y}{\partial x_1}=\frac{\partial f}{\partial u_1}\frac{\partial u_1}{\partial x_1}\Delta x_1+\frac{\partial f}{\partial u_2}\frac{\partial u_2}{\partial x_1}\Delta x_1...+\frac{\partial f}{\partial u_n}\frac{\partial u_n}{\partial x_1}\Delta x_1$

4、 多元函数的极值

多元函数的极值：极值点的梯度为0向量

$f(\vec{x})=f(\overrightarrow{x_0})+(\vec{x}-\overrightarrow{x_0})gradf(\overrightarrow{x_0})+0.5(\vec{x}-\overrightarrow{x_0})H(\vec{x}-\overrightarrow{x_0}{)}^T$

$H=\left(\begin{array}{cc}\frac{{\partial }^{2}f}{\partial x_1^2}& \frac{{\partial }^{2}f}{\partial x_1\partial x_2}\\ \frac{{\partial }^{2}f}{\partial x_2\partial x_1}& \frac{{\partial }^{2}f}{\partial x_2^2}\end{array}\right)$
$(\vec{x}-\overrightarrow{x_0})H(\vec{x}-\overrightarrow{x_0}{)}^T=(x_1-x_1^0,x_2-x_2^0)\left(\begin{array}{cc}\frac{{\partial }^{2}f}{\partial x_1^2}& \frac{{\partial }^{2}f}{\partial x_1\partial x_2}\\ \frac{{\partial }^{2}f}{\partial x_2\partial x_1}& \frac{{\partial }^{2}f}{\partial x_2^2}\end{array}\right)\left(\begin{array}{c}{x}_1-x_1^0\\ x_2-x_2^0\end{array}\right)=0.5[(x_1-x_1^0{)}^2\frac{{\partial }^{2}f}{\partial x_1^2}+2(x_1-x_1^0)(x_2-x_2^0)\frac{{\partial }^{2}f}{\partial x_2\partial x_1}+(x_2-x_2^0{)}^2\frac{{\partial }^{2}f}{\partial x_2^2}]=0.5[\Delta x_1^2\frac{{\partial }^{2}f}{\partial x_1^2}+2(\Delta x_1)(\Delta x_2)\frac{{\partial }^{2}f}{\partial x_2\partial x_1}+\Delta x_2^2\frac{{\partial }^{2}f}{\partial x_2^2}]$

H正定：$\forall \vec{v},\vec{v}H{\vec{v}}^T>0$，往任意方向都会变大，证明这点取极小值
H负定：$\forall \vec{v},\vec{v}H{\vec{v}}^T<0$，往任意方向都会变小，证明这点取极大值
H不定：$\exists ,\overrightarrow{v_1}\overrightarrow{v_1}H{\overrightarrow{v_1}}^T<0，\exists \overrightarrow{v_2},\overrightarrow{v_2}H{\overrightarrow{v_2}}^T>0$

5、拉格朗日乘数法

隐函数定理：
$f(x_1,...x_n)$
$\begin{array}{r}{\phi }_1(x_1,...x_n)=0\\ {\phi }_2(x_1,...x_n)=0\\ ...\\ {\phi }_m(x_1,...x_n)=0\end{array}\}\begin{array}{r}{x}_{n-m+1}=g_{n-m+1}(x_1,...x_{n-m})\\ x_{n-m+2}=g_{n-m+2}(x_1,...x_{n-m})\\ ...\\ x_n=g_n(x_1,...x_{n-m})\end{array}$可替换f中对应的变量，将求带条件的极值转化为求不带条件的极值

$f(x_1,...,x_n)=k(x_1,...,x_m)$

$\frac{\partial k}{\partial x_1}=\frac{\partial f}{\partial x_1}+\frac{\partial f}{\partial g_{n-m+1}}\frac{\partial g_{n-m+1}}{\partial x_1}+...+\frac{\partial f}{\partial g_n}\frac{\partial g_n}{\partial x_1}$

$(\frac{\partial f(\overrightarrow{x_0})}{\partial x_1},...,\frac{\partial f(\overrightarrow{x_0})}{\partial x_{n-m}})+(\frac{\partial f(\overrightarrow{x_0})}{\partial x_{n-m+1}},...,\frac{\partial f(\overrightarrow{x_0})}{\partial x_n})g^{\prime }({\overrightarrow{x_0}}^{\prime })=0$，第一项与前式$\frac{\partial f}{\partial x_1}$对应，余下项与前式余下项对应

$g^{\prime }({\overrightarrow{x_0}}^{\prime })=-{\left(\begin{array}{ccc}\frac{\partial {\phi }_1(x_0)}{\partial x_{n-m+1}}& ...& \frac{\partial {\phi }_1(x_0)}{\partial x_n}\\ ...& & ...\\ \frac{\partial {\phi }_m(x_0)}{\partial x_{n-m+1}}& ...& \frac{\partial {\phi }_m(x_0)}{\partial x_n}\end{array}\right)}^{-1}\left(\begin{array}{ccc}\frac{\partial {\phi }_1(x_0)}{\partial x_1}& ...& \frac{\partial {\phi }_1(x_0)}{\partial x_{n-m}}\\ ...& & ...\\ \frac{\partial {\phi }_m(x_0)}{\partial x_1}& ...& \frac{\partial {\phi }_m(x_0)}{\partial x_{n-m}}\end{array}\right)$

1*(n-m)维+(1m维)(m*m维)，可以通过$\frac{\partial x_{n-m+1}}{\partial {\phi }_1}\frac{\partial {\phi }_1}{\partial x_1}=\frac{\partial x_{n-m+1}}{\partial x_1}$理解$g^{\prime }({\overrightarrow{x_0}}^{\prime })$的构成。

$({\lambda }_1,{\lambda }_2,...,{\lambda }_m)=(\frac{\partial f(\overrightarrow{x_0})}{\partial x_{n-m+1}},...,\frac{\partial f(\overrightarrow{x_0})}{\partial x_n})\ast g^{\prime }({\overrightarrow{x_0}}^{\prime })的第一项=(\frac{\partial f(\overrightarrow{x_0})}{\partial x_{n-m+1}},...,\frac{\partial f(\overrightarrow{x_0})}{\partial x_n})\left(-{\left(\begin{array}{ccc}\frac{\partial {\phi }_1(x_0)}{\partial x_{n-m+1}}& ...& \frac{\partial {\phi }_1(x_0)}{\partial x_n}\\ ...& & ...\\ \frac{\partial {\phi }_m(x_0)}{\partial x_{n-m+1}}& ...& \frac{\partial {\phi }_m(x_0)}{\partial x_n}\end{array}\right)}^{-1}\right)$，$({\lambda }_1,{\lambda }_2,...,{\lambda }_m)$与$g^{\prime }({\overrightarrow{x_0}}^{\prime })$的第二项的第一列相乘有：

$\frac{\partial f}{\partial x_1}+{\lambda }_1\frac{\partial {\phi }_1}{\partial x_1}+{\lambda }_2\frac{\partial {\phi }_2}{\partial x_1}+...+{\lambda }_m\frac{\partial {\phi }_m}{\partial x_1}=0$

令$F=f+{\lambda }_1{\phi }_1+{\lambda }_2{\phi }_2+...+{\lambda }_m{\phi }_m$有$\frac{\partial F}{\partial x_i}=0$，i从1到n-m，

由$({\lambda }_1,{\lambda }_2,...,{\lambda }_m)=(\frac{\partial f(\overrightarrow{x_0})}{\partial x_{n-m+1}},...,\frac{\partial f(\overrightarrow{x_0})}{\partial x_n})\left(-{\left(\begin{array}{ccc}\frac{\partial {\phi }_1(x_0)}{\partial x_{n-m+1}}& ...& \frac{\partial {\phi }_1(x_0)}{\partial x_n}\\ ...& & ...\\ \frac{\partial {\phi }_m(x_0)}{\partial x_{n-m+1}}& ...& \frac{\partial {\phi }_m(x_0)}{\partial x_n}\end{array}\right)}^{-1}\right)$得$({\lambda }_1,{\lambda }_2,...,{\lambda }_m)\left(\begin{array}{ccc}\frac{\partial {\phi }_1(x_0)}{\partial x_{n-m+1}}& ...& \frac{\partial {\phi }_1(x_0)}{\partial x_n}\\ ...& & ...\\ \frac{\partial {\phi }_m(x_0)}{\partial x_{n-m+1}}& ...& \frac{\partial {\phi }_m(x_0)}{\partial x_n}\end{array}\right)+(\frac{\partial f(\overrightarrow{x_0})}{\partial x_{n-m+1}},...,\frac{\partial f(\overrightarrow{x_0})}{\partial x_n})=0$得：

$\frac{\partial f}{\partial x_{n-m+1}}+{\lambda }_1\frac{\partial {\phi }_1}{\partial x_{n-m+1}}+{\lambda }_2\frac{\partial {\phi }_2}{\partial x_{n-m+1}}+...+{\lambda }_m\frac{\partial {\phi }_m}{\partial x_{n-m+1}}=0$即为$\frac{\partial F}{\partial x_{n-m+1}}=0$

有$\frac{\partial F}{\partial x_i}=0$，i从n-m+1到n。f在点$\overrightarrow{x_0}、\vec{\lambda }$满足式子$\frac{\partial F}{\partial x_i}=0$（n个式子），同时满足条件${\phi }_1(\overrightarrow{x_0})={\phi }_2(\overrightarrow{x_0})=...=0$，在$\overrightarrow{x_0}$取极值。通过隐函数方法把条件带回去，将条件极值转化为非条件极值。