梯度下降法的一般步骤为:
(1) 设定初始值
(2) 求取梯度
(3) 在梯度方向上进行参数的更新
用numpy实现:
import numpy as np
import matplotlib.pyplot as plt
#目标函数
def func(x):
return x**2 + 1
#目标函数的梯度
def grad(x):
return x*2
# 目标函数图像
plot_x = np.linspace(-3,3,140)
plot_y = func(plot_x)
#梯度下降算法参数
learning_rate = 0.01
epsilon = 0.0001
max_iters = 10000
history_x = []
x = 10
for i in range(max_iters):
cur_grad = grad(x)
last_x = x
x = x - learning_rate * cur_grad
history_x.append(x)
if(abs(func(x)-func(last_x))<epsilon):
break
plt.plot(plot_x,plot_y)
plt.plot(np.array(history_x),func(np.array(history_x)),color = 'r',marker='+')
plt.show()
结果如图:
用pytorch实现梯度下降:
import torch
from torch.autogtad import Variable
x = torch.Tensor([1])
x = Variable(x,requires_grad = True)
print('grad',x.grad,'data',x.data)
learing_rate = 0.1
epochs = 10
for epoch in range(epochs):
y = x ** 2 + 2 * x + 1
y.backward()
print('grad',x.grad.data)
x.data = x.data - learing_rate * w.grad.data
x.grad.data.zero()
print(x.data)