手撕梯度下降

逻辑回归的梯度下降法

Gradient Descent for Logistic Regressian

$$\begin{gathered} P(y-1|x,w)=\frac{1}{1+e^{-w^{T}x+b}} \\ argmi{n}_{w,b}=-\sum _{i=1}^{n}y\log P(y=1|x,w)+(1-y)\log (1-P(y=1|x,w)) \end{gathered}$$

迭代式的方法

$$\begin{gathered} w^{t-1}=w^t-{\eta }_t{\sum }_{i=1}^n[\sigma (w^T{x}_i+b)-y_i]x_i \\ {b}^{t-1}=b^t-{\eta }_t{\sum }_{i=1}^n[\sigma (w^T{x}_i+b)-y_i] \end{gathered}$$

梯度下降法的复杂度

Q: What is the complexity of gradient descr

快排,归并排序时间复杂度$O(n\log n)$,那么梯度下降法呢,他是迭代的过程(需要多长时间保最优解,初始值,)

梯度下降法的收敛分析 Gradient Descent Algorithm

梯度下降法的过程可以表示为

• 选择初始值 $x_0\in R^d$, 和步长（step-size) ${\eta }_t>0$
• $fori=0,1,\dots$

$$x_{i+1}=x_iー\eta ▽f(x_i)$$

Convergence Analysis of Gradient Descent

定理¹

假设函数满足$ Lー Lipschitz$条件，并且是凸函数，设定$ x^*= argminf(x）$那么对于步长$ t\leq\frac{1}{L}$，满足
$$f(x_k)\le f(x^{\ast })+\frac{||x_0-x^{\ast }|{|}_2^2}{2{\eta }_{t}k}$$

当我们迭代 $k=\frac{L||x_0-x^{\ast }|{|}_2^2}{ϵ}$次之后我们可以保证得到$ϵ-approximationoptimalvaluex({\eta }_t=1/L)$

误差优化

A算法:$f(x_{10})\leq f(x^)+0.1\f(x_{100})\leq f(x^)+0.0001\$

B算法:$f(x_{10})\leq f(x^)+0.1\f(x_{100})\leq f(x^)+0.00000000000001\$

凸函数性质以及 L- Lipschitz 条件

$$f(x_k)\le f(x^{\ast })+\frac{||x_0-x^{\ast }|{|}_2^2}{2{\eta }_{t}k}k=\frac{L||x_0-x^{\ast }|{|}^2}{ϵ}$$

将$k=\frac{L||x_0-x^{\ast }|{|}^2}{ϵ}$带入原式,注 ${\eta }_t=\frac{1}{L}$
$$f(x_k)\le f(x^{\ast })+O(ϵ)$$

$f(x)$是convex function.任意$x,y\in R^d,0\le \lambda \le 1$,对于first ordey convexity $f(x)+▽f(x)(y-1)\le f(y)$

L- Lipschitz条件以及定理

一个光滑函数（smooth function)$f$满足 L- Lipschitz 条件，则对于任意$x,y\in R^d$，我们有
$$||▽f(x)-▽f(y)||\le L||x-y||$$

i.e: linear Regrssion

$L=\frac{1}{n}||Xw-y|{|}^2$

$$\begin{array}{rl}||▽f(x)-▽f(y)||& =\frac{2}{n}||x^T(x{w}_1-y)-x^T(x{w}_2)-y||\\ & =\frac{1}{n}||x^{T}x(w_1-w_2)||\\ & \le \frac{2}{n}\underset{L}{\underbrace{||x^{T}x||}}\cdot ||w_1-w_2||\end{array}$$

假设一个函数满足 L- Lipschittz 条件，并且是凸函数，对于任意 $x,y\in R^d$，我们有

$$f(y)\le f(x)+▽f(x)f(yーx)+\frac{L}{2}||y-x|{|}^2$$

$h(x):h(1)=h(0)+{\int }_0^1{h}^{‘}(\tau )d\tau$$

定义函数$h(\tau )=f(x+\tau (y-x))$

$h(1)=f(y),h(0)=f(x)$

$f(y)=f(x)+{\int }_0^1▽f(x+\tau (y-x))(y-x)d\tau$

$$\begin{array}{rl}f(y)& =f(x)+{\int }_0^1▽f(x+\tau (y-x))(y-x)d\tau \\ & =f(x)+▽f(x)(y-x)+{\int }_0^1(▽f(x+\tau (y-x))-▽f(x))(y-x)d\tau \\ & \le f(x)+▽f(x)(y-x)+{\int }_0^{1}L||\tau (y-x)||\cdot ||y-x||d\tau \\ & =f(x)+▽f(x)(y-x)+\frac{L}{2}||y-x|{|}^2\end{array}$$

$ab\le |a|+|b|$

Dervation

$$f(y)\le f(x)+▽f(x)(y-x)+\frac{L}{2}||y-x|{|}^2$$

那么使用$f(x_{i+1}),f(x_i)来进行证明$
$$\begin{array}{rl}f(x_{i+1})& \le f(x_i)+▽f(x_i)(x_{i+1}-x_i)+\frac{L}{2}||x_{i+1}-x_i|{|}^2\\ & =f(x_i)-▽f(x_i)\cdot (-1)\cdot {\eta }_t▽f(x_i)+\frac{L}{2}▽f(x_i{)}^2\\ & =f(x_i)-{\eta }_t||▽f(x_i)|{|}^2+\frac{L\cdot {\eta }_t^2}{2}||▽f(x_i)|{|}^2\\ & =f(x_i)={\eta }_t(1-\frac{L{\eta }_t}{2})||▽f(x_i)|{|}^2\\ & \le f(x_i)-\frac{{\eta }_t}{2}||▽f(x_i)|{|}^2\end{array}$$
注解

${\eta }_t\le \frac{1}{L}$

Derivation

$$\begin{array}{rl}f(x_{i+1})& \le f(x_i)-\frac{{\eta }_t}{2}||▽f(x_i)|{|}^2\\ & =f(x^{\ast })+▽f(x_i)(x_i-x^{\ast })-\frac{{\eta }_t}{2}||▽f(x_i)|{|}_2^2\\ & =f(x^{\ast })+\frac{x_i-x_{i+1}}{{\eta }_t}\cdot (x_i-x^{\ast })-\frac{1}{2{\eta }_t}||x_i-x_{i+1}|{|}^2\\ & =f(x^{\ast })+\frac{1}{2{\eta }_t}||x_i-x^{\ast }|{|}^2-\frac{1}{2{\eta }_t}(||x_i-x^{\ast }|{|}^2-2{\eta }_t▽f(x_i)(x_i-x^{\ast })+||{\eta }_t▽f(x_i)|{|}^2)\\ & =f(x^{\ast })+\frac{1}{2{\eta }_t}||x_i-x^{\ast }|{|}^2-\frac{1}{2{\eta }_t}||x_i-x^{\ast }-{\eta }_t▽f(x)i)|{|}^2\\ & =f(x^{\ast })+\frac{1}{2{\eta }_t}(||x_i-x^{\ast }|{|}^2-||x_{i+1}-x^{\ast }|{|}^2)\end{array}$$

注解

$$\begin{gathered} (a+b{)}^2=a^2+2ab+b^2 \\ {x}_{i+1}=x_i-{\eta }_t\cdot ▽f(x_i) \end{gathered}$$

Derivation

$$\begin{array}{rl}f(x_{i+1})& \le f(x^{\ast })+\frac{1}{2{\eta }_t}(||x_i-x^{\ast }|{|}^2-||x_{i+1}-x^{\ast }|{|}^2)\\ f(x_{i+1})-f(x^{\ast })& \le \frac{1}{2{\eta }_t}(||x_i-x^{\ast }|{|}^2-||x_{i+1}-x^{\ast }|{|}^2)\end{array}$$

袋鼠数据,急性绩效

$$\begin{gathered} f(x_1)-f(x^{\ast })\le \frac{1}{2{\eta }_t}(||x_0-x^{\ast }|{|}^2-||x_1-x^{\ast }|{|}^2) \\ f(x_2)-f(x^{\ast })\le \frac{1}{2{\eta }_t}(||x_1-x^{\ast }|{|}^2-||x_2-x^{\ast }|{|}^2) \\ f(x_3)-f(x^{\ast })\le \frac{1}{2{\eta }_t}(||x_2-x^{\ast }|{|}^2-||x_3-x^{\ast }|{|}^2) \\ ... \\ f(x_k)-f(x^{\ast })\le \frac{1}{2{\eta }_t}(||x_{k-1}-x^{\ast }|{|}^2-||x_k-x^{\ast }|{|}^2) \end{gathered}$$

相加之后
$$\begin{array}{rl}\sum _{i=1}^{k}f(x_i)-kf(x^{\ast })& \le \frac{1}{2{\eta }_t}(||x_0-x^{\ast }|{|}^2-||x_k-x^{\ast }|{|}^2)\\ \sum _{i=1}^{k}f(x_i)-k\cdot f(x^{\ast })& \le \frac{1}{2{\eta }_t}||x+0-x^{\ast }|{|}^2\end{array}$$

注解,在梯度下降之下才成立,否则不成立,尤其是随机梯度下降

$$\begin{gathered} f(x_{i+1})\le f(x_i)-\frac{{\eta }_t}{2}||▽f(x_i)|{|}^2 \\ f(x_{i+1})\le f(x_i) \\ f(x_k)\le f(x_{k-1})\le f(x_{k-2})\le f(x_{k-3})\le f(x_0) \\ k\cdot f(x_k)-kf(x^{\ast })\le {\sum }_{i=1}^{k}f(x_i)-kf(x^{\ast })\le \frac{1}{2{\eta }_t}||x_0-x^{\ast }|{|}^2 \\ f(x_k)-f(x^{\ast })\le \frac{1}{2{\eta }_{t}k}||x_0-x^{\ast }|{|}^2 \end{gathered}$$

这个歌问题是Stocklast

收敛性推导

Linear classifier

D = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . . , ( x n , y n ) , } , y i ∈ { − 1 , 1 } , i = 1 , 2 , . . . , n D=\{(x_1,y_1),(x_2,y_2),...,(x_n,y_n),\},y_i\in \{-1,1\},i=1,2,...,n D={(x1​,y1​),(x2​,y2​),...,(xn​,yn​),},yi​∈{−1,1},i=1,2,...,n

对于线性模型的参数 θ = { w , b } , w T ⋅ x + b = 0 \theta =\{w,b\},w^T·x+b=0 θ={w,b},wT⋅x+b=0

$$\begin{gathered} w^T{x}_i+b\ge 0.y_i=1 \\ {w}^T{x}_i+b\le 0.y_i=-1 \end{gathered}$$

上面的可以写成
$$(w^T{x}_i+b)\cdot y_i\ge 0$$

错误还请大佬们指出,推导不一定正确,爱好,纯属爱好,记录美好生活,从点滴做起!

---

1. 注意两个条件 ↩︎