线性回归是监督学习最简单的一种模型了,在了解单特征的回归模型的基础上,再看较之复杂一些的多特征模型。也就是原来的例子是房子的大小对房价的影响,现在变为房子的大小、房间数目、层数、使用年限对房价的影响。由一个特征变成了多个特征。
Linear Regression with Multiple Features
这里的model用到了向量点乘
在算法中使用向量可以提高效率,下面介绍了向量化
关于模型构建的算法,上面列举了三种:
第一种:写起来复杂,运行慢
第二种:利用循环使得写起来简单了,但运行同样费时间
第三种:利用numpy中的np.dot点乘来完成,一行就完成,是并行计算的,速度快
此外,在梯度下降更新参数时,设学习率为0.1,用向量的计算(w=w-0.1*d)同样也更快
怎么来量化这个向量比循坏快呢?看算法:
def my_dot(a, b):
"""
Compute the dot product of two vectors
Args:
a (ndarray (n,)): input vector
b (ndarray (n,)): input vector with same dimension as a
Returns:
x (scalar):
"""
x=0
for i in range(a.shape[0]):
x = x + a[i] * b[i]
return x
import time
np.random.seed(1)
a = np.random.rand(10000000) # very large arrays
b = np.random.rand(10000000)
tic = time.time() # capture start time
c = np.dot(a, b)
toc = time.time() # capture end time
print(f"np.dot(a, b) = {c:.4f}")
print(f"Vectorized version duration: {1000*(toc-tic):.4f} ms ")
tic = time.time() # capture start time
c = my_dot(a,b)
toc = time.time() # capture end time
print(f"my_dot(a, b) = {c:.4f}")
print(f"loop version duration: {1000*(toc-tic):.4f} ms ")
del(a);del(b) #remove these big arrays from memory
np.dot耗时8.6598毫秒,而循环耗时2756.6850毫秒
梯度下降
下面是这部分的算法:
1.用到的工具
import copy, math
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('./deeplearning.mplstyle')
np.set_printoptions(precision=2) # reduced display precision on numpy arrays
2.特征和训练样本集
这里的是房子的大小,
是房间的数量,
是房屋的层数,
是房屋的已使用年限。4个特征,n为4。这里给出的训练样本有三个。
X_train = np.array([[2104, 5, 1, 45], [1416, 3, 2, 40], [852, 2, 1, 35]])
y_train = np.array([460, 232, 178])
# data is stored in numpy array/matrix
print(f"X Shape: {X_train.shape}, X Type:{type(X_train)})")
print(X_train)
print(f"y Shape: {y_train.shape}, y Type:{type(y_train)})")
print(y_train)
2.初始化模型参数 W(向量) 和 b(数值)
这里,n=4
b_init = 785.1811367994083
w_init = np.array([ 0.39133535, 18.75376741, -53.36032453, -26.42131618])
print(f"w_init shape: {w_init.shape}, b_init type: {type(b_init)}")
3. 计算f(X)=w·x+b——点乘用循环计算
def predict_single_loop(x, w, b):
"""
single predict using linear regression
Args:
x (ndarray): Shape (n,) example with multiple features
w (ndarray): Shape (n,) model parameters
b (scalar): model parameter
Returns:
p (scalar): prediction
"""
n = x.shape[0]
p = 0
for i in range(n):
p_i = x[i] * w[i]
p = p + p_i
p = p + b
return p
# get a row from our training data
x_vec = X_train[0,:]
print(f"x_vec shape {x_vec.shape}, x_vec value: {x_vec}")
# make a prediction
f_wb = predict_single_loop(x_vec, w_init, b_init)
print(f"f_wb shape {f_wb.shape}, prediction: {f_wb}")
将第一组样本[2104 5 1 45]代入模型,计算预测值459.9999976194083
4.更快的方法计算f(X)=w·x+b——np.dot
def predict(x, w, b):
"""
single predict using linear regression
Args:
x (ndarray): Shape (n,) example with multiple features
w (ndarray): Shape (n,) model parameters
b (scalar): model parameter
Returns:
p (scalar): prediction
"""
p = np.dot(x, w) + b #就这一行就OK
return p
# get a row from our training data
x_vec = X_train[0,:]
print(f"x_vec shape {x_vec.shape}, x_vec value: {x_vec}")
# make a prediction
f_wb = predict(x_vec,w_init, b_init)
print(f"f_wb shape {f_wb.shape}, prediction: {f_wb}")
结果和上面一样,只是运算更快了
5.成本函数,计算成本
def compute_cost(X, y, w, b):
"""
compute cost
Args:
X (ndarray (m,n)): Data, m examples with n features
y (ndarray (m,)) : target values
w (ndarray (n,)) : model parameters
b (scalar) : model parameter
Returns:
cost (scalar): cost
"""
m = X.shape[0] #m是训练样本集中样本个数
cost = 0.0
for i in range(m):
f_wb_i = np.dot(X[i], w) + b #(n,)(n,) = scalar (see np.dot)
cost = cost + (f_wb_i - y[i])**2 #scalar
cost = cost / (2 * m) #scalar
return cost
# Compute and display cost using our pre-chosen optimal parameters.
cost = compute_cost(X_train, y_train, w_init, b_init)
print(f'Cost at optimal w : {cost}')
6.多特征的梯度下降法
(1)计算偏导
def compute_gradient(X, y, w, b):
"""
Computes the gradient for linear regression
Args:
X (ndarray (m,n)): Data, m examples with n features
y (ndarray (m,)) : target values
w (ndarray (n,)) : model parameters
b (scalar) : model parameter
Returns:
dj_dw (ndarray (n,)): The gradient of the cost w.r.t. the parameters w.
dj_db (scalar): The gradient of the cost w.r.t. the parameter b.
"""
m,n = X.shape #(number of examples, number of features) m是样本数量,n是特征数量
dj_dw = np.zeros((n,)) #一维向量,全是0,一共有n个
dj_db = 0.
for i in range(m): #m个样本
err = (np.dot(X[i], w) + b) - y[i]
for j in range(n): #n个特征
dj_dw[j] = dj_dw[j] + err * X[i, j]
dj_db = dj_db + err
dj_dw = dj_dw / m
dj_db = dj_db / m
return dj_db, dj_dw
#Compute and display gradient
tmp_dj_db, tmp_dj_dw = compute_gradient(X_train, y_train, w_init, b_init)
print(f'dj_db at initial w,b: {tmp_dj_db}')
print(f'dj_dw at initial w,b: \n {tmp_dj_dw}')
通过以上函数,求w1,w2,w3,w4和b关于成本函数的偏导值
此外,我自己写了另一种循环方法求各个偏导值:
def compute_gradient(X, y, w, b):
m,n = X.shape #(number of examples, number of features) m是样本数量,n是特征数量
dj_dw = np.zeros((n,)) #一维向量,全是0,一共有n个
dj_db = 0.
for i in range(n): #n个特征
for j in range(m): #m个样本
err=(np.dot(X[j],w)+b)-y[j]
dj_dw[i]=dj_dw[i]+err*X[j,i]
dj_db=dj_db+err
dj_dw=dj_dw/m
dj_db = dj_db/n/m
return dj_db, dj_dw
结果和之前那个一模一样,但显然这个循环不如上面的好
(2)实现梯度下降
def gradient_descent(X, y, w_in, b_in, cost_function, gradient_function, alpha, num_iters):
"""
Performs batch gradient descent to learn theta. Updates theta by taking
num_iters gradient steps with learning rate alpha
Args:
X (ndarray (m,n)) : Data, m examples with n features
y (ndarray (m,)) : target values
w_in (ndarray (n,)) : initial model parameters
b_in (scalar) : initial model parameter
cost_function : function to compute cost
gradient_function : function to compute the gradient
alpha (float) : Learning rate
num_iters (int) : number of iterations to run gradient descent
Returns:
w (ndarray (n,)) : Updated values of parameters
b (scalar) : Updated value of parameter
"""
# An array to store cost J and w's at each iteration primarily for graphing later
J_history = []
w = copy.deepcopy(w_in) #avoid modifying global w within function
b = b_in
for i in range(num_iters):
# Calculate the gradient and update the parameters
dj_db,dj_dw = gradient_function(X, y, w, b) ##None
# Update Parameters using w, b, alpha and gradient
w = w - alpha * dj_dw ##None
b = b - alpha * dj_db ##None
# Save cost J at each iteration
if i<100000: # prevent resource exhaustion
J_history.append( cost_function(X, y, w, b))
# Print cost every at intervals 10 times or as many iterations if < 10
if i% math.ceil(num_iters / 10) == 0:
print(f"Iteration {i:4d}: Cost {J_history[-1]:8.2f} ")
return w, b, J_history #return final w,b and J history for graphing
上面函数中的 w = w - alpha * dj_dw 是向量计算(并行的,运算快)
# initialize parameters
initial_w = np.zeros_like(w_init) #构造一个矩阵initial_w,其维度数和w_init一样,值全为0
initial_b = 0.
# some gradient descent settings
iterations = 1000
alpha = 5.0e-7
# run gradient descent
w_final, b_final, J_hist = gradient_descent(X_train, y_train, initial_w, initial_b,
compute_cost, compute_gradient,
alpha, iterations)
print(f"b,w found by gradient descent: {b_final:0.2f},{w_final} ")
m,_ = X_train.shape
for i in range(m):
print(f"prediction: {np.dot(X_train[i], w_final) + b_final:0.2f}, target value: {y_train[i]}")
可以看出,成本在降低,最终获得的W=[0.2 0 -0.01 -0.07],b=0
最后有计算了三个样本的模型预测值和真实值
7.可视化J(绘制成本J和训练次数之间的图像)
# plot cost versus iteration
fig, (ax1, ax2) = plt.subplots(1, 2, constrained_layout=True, figsize=(12, 4))
ax1.plot(J_hist) #J_hist因变量,自变量默认从0开始取整数
ax2.plot(100 + np.arange(len(J_hist[100:])), J_hist[100:]) #自变量从100开始取到1000,因变量J_hist[100:]
ax1.set_title("Cost vs. iteration"); ax2.set_title("Cost vs. iteration (tail)")
ax1.set_ylabel('Cost') ; ax2.set_ylabel('Cost')
ax1.set_xlabel('iteration step') ; ax2.set_xlabel('iteration step')
plt.show()
These results are not inspiring! Cost is still declining and our predictions are not very accurate. The next lab will explore how to improve on this.
也就是说,梯度下降时,设置了迭代1000次,但这1000次之后成本还在下降,还未到达局部最小值,还未收敛。