机器学习学习记录2 - 逻辑回归
问题提出:
构建逻辑回归模型预测某个学生是否被大学录取
已知条件:
对于每一个训练样本有两次测试的评分和最终的录取结果
1.构建sigmoid函数
g代表一个常用的逻辑函数为S形函数,公式为:
g
(
z
)
=
1
1
+
e
−
z
g(z)=\frac{1}{1+e^{-z}}
g(z)=1+e−z1
结合之前的假设函数,我们可以得到逻辑回归模型的假设函数:
h
θ
(
x
)
=
1
1
+
e
−
θ
T
X
h_\theta(x)=\frac{1}{1+e^{-\theta^TX}}
hθ(x)=1+e−θTX1
#通过Python定义sigmoid函数
def sigmoid(z):
return 1 / (1 + np.exp(-z))
#sigmoid函数可视化
nums = np.arange(-10, 10, step=1)#返回一个
#print(nums)
fig, ax = plt.subplots(figsize=(12,8))
ax.plot(nums, sigmoid(nums), 'r')
plt.show()
2.编写代价函数
代价函数:
J
(
θ
)
=
1
m
∑
i
=
1
m
[
−
y
(
i
)
l
o
g
(
h
θ
(
x
(
i
)
)
)
−
(
1
−
y
(
i
)
)
l
o
g
(
1
−
h
θ
(
x
(
i
)
)
)
]
J(\theta)=\frac{1}{m}\sum_{i=1}^{m}[-y^{(i)}log(h_{\theta}(x^{(i)}))-(1-y^{(i)})log(1-h_{\theta}(x^{(i)}))]
J(θ)=m1i=1∑m[−y(i)log(hθ(x(i)))−(1−y(i))log(1−hθ(x(i)))]
#编写代价函数
def cost(theta, X, y):
theta = np.matrix(theta)
X = np.matrix(X)
y = np.matrix(y)
first = np.multiply(-y, np.log(sigmoid(X * theta.T)))#np.multiply 两个参数相乘
second = np.multiply((1 - y), np.log(1 - sigmoid(X * theta.T)))
return np.sum(first - second) / (len(X))
3.导入数据并将其可视化
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
path = 'ex2data1.txt'
data = pd.read_csv(path, header=None, names=['Exam 1', 'Exam 2', 'Admitted'])
data.head()
#通过创建散点图表示正样本和负样本
positive = data[data['Admitted'].isin([1])]
negative = data[data['Admitted'].isin([0])]
fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['Exam 1'], positive['Exam 2'], s=50, c='b', marker='o', label='Admitted')
ax.scatter(negative['Exam 1'], negative['Exam 2'], s=50, c='r', marker='x', label='Not Admitted')
ax.legend()
ax.set_xlabel('Exam 1 Score')
ax.set_ylabel('Exam 2 Score')
plt.show()
通过对数据可视化我们可以看到正负样本之间有清晰的边界,因此可以通过逻辑回归来实现对结果预测
#添加偏置项
# add a ones column - this makes the matrix multiplication work out easier
data.insert(0, 'Ones', 1)#第0列添加偏置为1的标签项Ones
# set X (training data) and y (target variable)
cols = data.shape[1]
X = data.iloc[:,0:cols-1]
y = data.iloc[:,cols-1:cols]
# convert to numpy arrays and initalize the parameter array theta
X = np.array(X.values)
y = np.array(y.values)
theta = np.zeros(3)
可以通过调用cost函数
计算代价
#此时初始化参数theat为0
cost(theat,X,y)
4.gradient descent(梯度下降)
对于批量梯度下降(BGD)转换为向量化计算得到:
∂
J
(
θ
)
∂
θ
j
=
1
m
∑
i
=
1
m
(
h
θ
(
x
(
i
)
)
−
y
(
i
)
)
x
j
i
\frac{\partial J(\theta)}{\partial\theta_j}=\frac{1}{m}\sum_{i=1}^{m}(h_\theta(x^{(i)})-y^{(i)})x^{i}_j
∂θj∂J(θ)=m1i=1∑m(hθ(x(i))−y(i))xji
def gradient(theta, X, y):
theta = np.matrix(theta)
X = np.matrix(X)
y = np.matrix(y)
parameters = int(theta.ravel().shape[1])#ravel()多维转一维 shape[1]是取矩阵1*3中的3
grad = np.zeros(parameters)
error = sigmoid(X * theta.T) - y
for i in range(parameters):
term = np.multiply(error, X[:,i])
grad[i] = np.sum(term) / len(X)
return grad
gradient(theata,X,y)
该代码块仅仅执行一个梯度步长,接下来通过Scipy来寻找最优参数
import scipy.optimize as opt
#该函数用来寻找最优参数及theat的最优解
result = opt.fmin_tnc(func=cost, x0=theta, fprime=gradient, args=(X, y))
result
#(array([-25.16131863, 0.20623159, 0.20147149]), 36, 0) result的结果
cost(result[0], X, y)#result[0]及就是优化后的theta
#结果:0.20349770158947458
5.预测
编写函数通过所学习的参数
θ
\theta
θ对数据集X进行预测,同时为我们的分类器的训练精度打分,逻辑回归模型的假设函数为:
h
θ
(
x
)
=
1
1
+
e
−
θ
T
X
h_\theta(x)=\frac{1}{1+e^{-\theta^{T}X}}
hθ(x)=1+e−θTX1
当
h
θ
h_\theta
hθ大于等于0.5时,预测y=1
当 h θ h_\theta hθ小于0.5时,预测y=0.
def predict(theta, X):
probability = sigmoid(X * theta.T)
return [1 if x >= 0.5 else 0 for x in probability]
theta_min = np.matrix(result[0])
predictions = predict(theta_min, X)
#zip是将对象中的元素打包成元组形式#判断predictions与y是否相等,若相等返回1否则返回0
correct = [1 if ((a == 1 and b == 1) or (a == 0 and b == 0)) else 0 for (a, b) in zip(predictions, y)]
#计算准确率,用correct中两个元素相等的个数除以总个数也就是准确率。
accuracy = (sum(map(int, correct)) % len(correct))
print ('accuracy = {0}%'.format(accuracy))
#结果输出准确率为89%
因此得到了预测分类器的准确率为89%,之后将介绍使用交叉验证进行真实逼近
6.正则化逻辑回归
加入正则项有助于减少过拟合,提高模型的泛化能力
正则化代价函数
J ( θ ) = 1 m ∑ i = 1 m [ − y ( i ) l o g ( h θ ( x ( i ) ) ) − ( 1 − y ( i ) ) l o g ( 1 − h θ ( x ( i ) ) ) ] + λ 2 m ∑ j = 1 n θ j 2 J(\theta)=\frac{1}{m}\sum_{i=1}^{m}[-y^{(i)}log(h_{\theta}(x^{(i)}))-(1-y^{(i)})log(1-h_{\theta}(x^{(i)}))]+\frac{\lambda}{2m}\sum_{j=1}^{n}\theta^2_j J(θ)=m1i=1∑m[−y(i)log(hθ(x(i)))−(1−y(i))log(1−hθ(x(i)))]+2mλj=1∑nθj2
def costReg(theta, X, y, learningRate):
theta = np.matrix(theta)
X = np.matrix(X)
y = np.matrix(y)
first = np.multiply(-y, np.log(sigmoid(X * theta.T)))
second = np.multiply((1 - y), np.log(1 - sigmoid(X * theta.T)))#其他和上述保持一致
reg = (learningRate / (2 * len(X))) * np.sum(np.power(theta[:,1:theta.shape[1]], 2))#正则化项,不能正则化第一项theta0(theta的第一列)
return np.sum(first - second) / len(X) + reg
未对 θ \theta θ项进行正则化,因此,更新式子变为:
当j=1,2,…,n时候可得:
θ
j
:
=
θ
j
(
1
−
α
λ
m
)
−
α
1
m
∑
i
=
1
m
(
h
θ
(
x
(
i
)
)
−
y
(
i
)
)
x
j
(
i
)
\theta_j:=\theta_j(1-\alpha\frac{\lambda}{m})-\alpha\frac{1}{m}\sum_{i=1}^{m}(h_\theta(x^{(i)})-y^{(i)})x^{(i)}_j
θj:=θj(1−αmλ)−αm1i=1∑m(hθ(x(i))−y(i))xj(i)
#实现代码如下
def gradientReg(theta, X, y, learningRate):
theta = np.matrix(theta)
X = np.matrix(X)
y = np.matrix(y)
parameters = int(theta.ravel().shape[1])#将theta变为一维且得到其列数
grad = np.zeros(parameters)
error = sigmoid(X * theta.T) - y
for i in range(parameters):
term = np.multiply(error, X[:,i])
if (i == 0):
grad[i] = np.sum(term) / len(X)
else:
grad[i] = (np.sum(term) / len(X)) + ((learningRate / len(X)) * theta[:,i])
return grad
接着像之前一样初始化变量
# set X and y (remember from above that we moved the label to column 0)
cols = data2.shape[1]
X2 = data2.iloc[:,1:cols]
y2 = data2.iloc[:,0:1]
# convert to numpy arrays and initalize the parameter array theta
X2 = np.array(X2.values)
y2 = np.array(y2.values)
theta2 = np.zeros(11)
选择一个初始学习率和初始theta(为0)
learningRate = 1
costReg(theta2, X2, y2, learningRate)
#结果:0.6931471805599454
#执行一次梯度下降结果
gradientReg(theta2, X2, y2, learningRate)
#使用和第一部分一样的优化函数计算优化结果
result2 = opt.fmin_tnc(func=costReg, x0=theta2, fprime=gradientReg, args=(X2, y2, learningRate))
result2
(array([ 0.53010247, 0.29075567, -1.60725764, -0.58213819, 0.01781027,
-0.21329507, -0.40024142, -1.3714414 , 0.02264304, -0.95033581,
0.0344085 ]),
22,
1)
#使用与第一部分同样的方法来监测准确度
theta_min = np.matrix(result2[0])
predictions = predict(theta_min, X2)
correct = [1 if ((a == 1 and b == 1) or (a == 0 and b == 0)) else 0 for (a, b) in zip(predictions, y2)]
accuracy = (sum(map(int, correct)) % len(correct))
print ('accuracy = {0}%'.format(accuracy))
#最终准确率为78%
7.scikit-learn方法解决
from sklearn import linear_model#调用sklearn的线性回归包
model = linear_model.LogisticRegression(penalty='l2', C=1.0)
model.fit(X2, y2.ravel())#ravel多维转一维
model.score(X2, y2)
#结果为0.6610169491525424
注:
我们可以看到这里的精度为0.66,比之前的0.78差了好多,但是该结果是在默认参数下计算的结果,可以通过调整参数获得与之前相同的精度。