梯度下降法
梯度下降法也叫做最速下降法(Steepest Descent Method),经常用来求解无约束优化的最小值问题。
对于函数 f ( x ) f(x) f(x),如果 f ( x ) f(x) f(x)在点 x t x_t xt附近是连续可微的,那么 f ( x ) f(x) f(x)下降最快的方向是 f ( x ) f(x) f(x)在 x t x_t xt点的梯度方向的反方向。
根据一阶泰勒展开式有:
f
(
x
t
+
1
)
=
f
(
x
t
+
Δ
x
)
≈
f
(
x
t
)
+
△
x
T
▽
f
(
x
t
)
f\left( x_{t+1} \right) =f\left( x_t+\varDelta x \right) \approx f\left( x_t \right) +\vartriangle x^T\triangledown f\left( x_t \right)
f(xt+1)=f(xt+Δx)≈f(xt)+△xT▽f(xt)
要使得
f
(
x
t
+
1
)
<
f
(
x
t
)
f(x_{t+1})<f(x_t)
f(xt+1)<f(xt),就使得
△
x
T
▽
f
(
x
t
)
<
0
\vartriangle x^T\triangledown f\left( x_t \right) <0
△xT▽f(xt)<0我们取
△
x
=
−
α
▽
f
(
x
t
)
\vartriangle x=-\alpha \triangledown f\left( x_t \right)
△x=−α▽f(xt)如果
α
>
0
\alpha >0
α>0为一个够小数值时,那么
f
(
x
t
+
1
)
<
f
(
x
t
)
f\left( x_{t+1} \right) <f\left( x_t \right)
f(xt+1)<f(xt)成立。
这样我们就可以从一个初值
x
0
x_0
x0出发,通过迭代公式
x
t
+
1
=
x
t
−
α
t
▽
f
(
x
t
)
,
t
≥
0
x_{t+1}=x_t-\alpha _t\triangledown f\left( x_t \right) ,t\ge 0
xt+1=xt−αt▽f(xt),t≥0
生成序列
x
0
,
x
1
,
x
2
,
⋅
⋅
⋅
x_0,x_1,x_2,···
x0,x1,x2,⋅⋅⋅使得
f
(
x
0
)
≥
f
(
x
1
)
≥
f
(
x
2
)
≥
f
(
x
3
)
⋅
⋅
⋅
⋅
f\left( x_0 \right) \ge f\left( x_1 \right) \ge f\left( x_2 \right) \ge f\left( x_3 \right) ····
f(x0)≥f(x1)≥f(x2)≥f(x3)⋅⋅⋅⋅
注:每次迭代步长
α
\alpha
α可以改变,但其取值必须合适,如果过大就不会收敛,如果过小则收敛的速度太慢。
随机梯度下降法的算法计算过程如下图[邱锡鹏nndl]:
这里我试着编写一下随机梯度下降法的程序~~(头大)~~ :
import numpy as np #导入numpy库
import matplotlib.pyplot as plt
import math
import random
train_dataset = np.loadtxt('train.txt')
test_dataset = np.loadtxt('test.txt')
#print(len(train_dataset))
#print(len(test_dataset))
#print(train_dataset)
#print(test_dataset)
# 随机初始化参数w
#w = np.zeros(2)#初始化系数
#w = np.mat(w)
#w = w.T
w = random.random()
b = random.random()
#print(w)
#print(train_dataset)
#print(train_dataset)
#train_y_dataset = np.mat(train_dataset[:,1])
#print(test_dataset)
test_y_dataset = test_dataset[:,1]
test_x_dataset = test_dataset[:,0]
#print(test_x_dataset)
array = train_dataset[:,1]
#print(array[1])
s = 0.0001 #初始化步长
sum_L = 0
#计算初始化时候产生的偏差
for i in range(len(test_dataset)):
L = math.pow(train_y_dataset[i]-(test_x_dataset[i]*w+b),2)
sum_L = L+sum_L
print(sum_L)
sum_l = 0
flag = 1
for i in range(100):
if(flag == 1):
for i in range(len(train_dataset)):
#print(i)
r = np.random.randint(0,300)
#print(r)
w = w + s*w*(array_y[r]-w*array_x[r]-b)
b = b + s*(array_y[r]-w*array_x[r]-b)
#print(w)
#print(b)
sum_l = 0
for j in range(len(test_dataset)):
l = math.pow(train_y_dataset[j]-(test_x_dataset[j]*w+b),2)#计算并更新错误率
sum_l = l + sum_l
#print(sum_l)
if(sum_l<sum_L):
sum_L=sum_l
#print(sum_L)
flag = 1
else:
flag = 0
print(sum_l)
if(flag==0):
break
print(w)
print(b)
#print(w.shape)
44563.115183671
953.7500949759108
0.9831895810000003
0.08080000000000129
#重新读取数据
x_train_dataset=train_dataset[:,0]
y_train_dataset=train_dataset[:,1]
#画散点图
fig = plt.figure()
x = np.linspace(0.01,30, 1000)
ax = fig.add_subplot(1,1,1)
ax.scatter(x_train_dataset,y_train_dataset,color='red',marker='o',s=1)
ax.plot(x, w*x+b,color='green',ls='-')
#画直线图
x_test_dataset=test_dataset[:,0]
#print(y_test_dataset)
x_test_dataset=np.mat(x_test_dataset).reshape(200,1)
#print(y_test_dataset)
y_test_dataset=test_dataset[:,1]
y_test_dataset=np.mat(y_test_dataset).reshape(200,1)
#print(y_test_dataset)
bias2 = 0
#测试
#bias2=bias2+math.pow((x_test_dataset[1]*0.94732599+0.62680273)-y_test_dataset[1],2)
#print(bias2)
for i in range (0,200):
bias2=bias2+math.pow((x_test_dataset[i]*w+b)-y_test_dataset[i],2)
print(bias2)
953.7500949759108