python实现吴恩达机器学习练习2（逻辑回归）-data2

python实现吴恩达机器学习练习2（逻辑回归）-data2

接上一篇：python实现吴恩达机器学习练习2（逻辑回归）-data1
假如对一个零件有两个测试指标，根据两个指标数值来决定是否为合格产品。我们用逻辑回归方法加入多项式项来拟合非线性决策边界。
参考链接：https://blog.csdn.net/Cowry5/article/details/80247569

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline

2 Regularized logistic regression

2.1 visualizing the data

data2 = pd.read_csv('D:/Python/exercise/samples/AndrewNg/ex2/ex2data2.csv', names = ['x1','x2','y'])

len(data2)

118

positive = data2[data2['y'] == 1]
negative = data2[data2['y'] == 0]
positive_x1 = positive['x1']
positive_x2 = positive['x2']
negative_x1 = negative['x1']
negative_x2 = negative['x2']

plt.figure(figsize = (10,10))
plt.scatter(x = positive_x1, y = positive_x2, marker = 'x', color = 'r', label = 'positive')
plt.scatter(x = negative_x1, y = negative_x2, color = 'lightgreen', label = 'negative')
plt.legend()

2.2 feature mapping

用二元数据生成指定次数的多项式

def polynomial_generator(x1, x2, power):
    data = {}
    for i in range(power + 1):
        for j in range(i + 1):
            data[f'{j}{i-j}'] = (x2**j) * (x1**(i-j))
    return data

用6次多项式

data = polynomial_generator(x1 = data2['x1'].values, x2 = data2['x2'].values, power = 6)

data_trans = pd.DataFrame(data)

data_trans.shape

(118, 28)

data_trans.head()

	00	01	10	02	11	20	03	12	21	30	...	32	41	50	06	15	24	33	42	51	60
0	1.0	0.051267	0.69956	0.002628	0.035864	0.489384	0.000135	0.001839	0.025089	0.342354	...	0.000900	0.012278	0.167542	1.815630e-08	2.477505e-07	0.000003	0.000046	0.000629	0.008589	0.117206
1	1.0	-0.092742	0.68494	0.008601	-0.063523	0.469143	-0.000798	0.005891	-0.043509	0.321335	...	0.002764	-0.020412	0.150752	6.362953e-07	-4.699318e-06	0.000035	-0.000256	0.001893	-0.013981	0.103256
2	1.0	-0.213710	0.69225	0.045672	-0.147941	0.479210	-0.009761	0.031616	-0.102412	0.331733	...	0.015151	-0.049077	0.158970	9.526844e-05	-3.085938e-04	0.001000	-0.003238	0.010488	-0.033973	0.110047
3	1.0	-0.375000	0.50219	0.140625	-0.188321	0.252195	-0.052734	0.070620	-0.094573	0.126650	...	0.017810	-0.023851	0.031940	2.780914e-03	-3.724126e-03	0.004987	-0.006679	0.008944	-0.011978	0.016040
4	1.0	-0.513250	0.46564	0.263426	-0.238990	0.216821	-0.135203	0.122661	-0.111283	0.100960	...	0.026596	-0.024128	0.021890	1.827990e-02	-1.658422e-02	0.015046	-0.013650	0.012384	-0.011235	0.010193

5 rows × 28 columns

2.3 cost function and gradient

#定义各个函数
def sigmoid(z):
    g = 1 / (1 + np.exp(-z))
    return g

def J_func(theta, x, y):
    cost = -y * np.log(sigmoid(x.dot(theta.T))) - (1-y) * np.log(1-sigmoid(x.dot(theta.T)))
    J = cost.mean()
    return J

$J(\theta )$加入正则项，$$J(\theta )=-\frac{1}{m}[\sum _{i=1}^m{y}^{(i)}ln{h}_{\theta }(x^{(i)})+(1-y^{(i)})ln(1-h_{\theta }(x^{(i)}))]+\frac{\lambda }{2m}\sum _{j=1}^n{\theta }_j^2$$

def J_func_reg(theta, x, y, c=1):
    _theta = theta[1:]
    reg = (c/(2*len(x)))*(_theta.dot(_theta.T))
    return J_func(theta, x, y) + reg

def gradient(theta, x, y):
    gra = x.T.dot(sigmoid(x.dot(theta.T))-y) / len(x)
    return gra    

改变梯度下降公式Gradient descent(因为${\theta }_0$不需要正则化，所以): r e p e a t { repeat\{ repeat{ $${\theta }_0:={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m[h_{\theta }(x^{(i)})-y^{(i)}]x_0^{(i)}$$ θ j : = θ j − α { 1 m ∑ i = 1 m [ h θ ( x ( i ) ) − y ( i ) ] x j ( i ) + λ m θ j } \theta_j := \theta_j-\alpha\{\frac{1}{m}\sum_{i=1}^m[h_{\theta}(x^{(i)})-y^{(i)}]x_j^{(i)}+\frac{\lambda}{m}\theta_j\} θj​:=θj​−α{m1​i=1∑m​[hθ​(x(i))−y(i)]xj(i)​+mλ​θj​} $$\}$$
gradient项变为：$$\frac{\partial }{\partial {\theta }_j}J(\theta )=\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}+\frac{\lambda }{m}{\theta }_j$$ $$(j=1,2,3,...,n)$$

def gradient_reg(theta, x, y, c=1):
    reg = (c/len(x)) * theta
    reg[0] = 0
    return gradient(theta, x, y) + reg

X = data_trans.values
Y = data2.iloc[:,-1].values
theta = np.zeros(28)

2.3.1 learning parameters using fmin_tnc

import scipy.optimize as opt

用两个$\lambda$值看看运行结果的区别

lambda_01 = 0.1
lambda_02 = 10

result = opt.fmin_tnc(func = J_func_reg, x0 = theta, fprime = gradient_reg, args = (X, Y, lambda_01)) #第三个参数为各方法的第四个参数
result_2 = opt.fmin_tnc(func = J_func_reg, x0 = theta, fprime = gradient_reg, args = (X, Y, lambda_02))

result = result[0]
result_2 = result_2[0]

2.4 plotting the decision boundary

a = np.arange(-1, 1.2, 0.01)
b = np.arange(-1, 1.2, 0.01)
xs, ys = np.meshgrid(a, b)

用ravel()方法把xs和ys数组扁平化，用polynomial_generator生成“列”为各种二项式组合，“行”为各x,y网格点组合的“字典”，然后再reshape回来

c_dict = polynomial_generator(xs.ravel(), ys.ravel(), 6)
c = pd.DataFrame(c_dict)
z = c.values.dot(result.T).reshape(xs.shape)
z_2 = c.values.dot(result_2.T).reshape(xs.shape)

fig, ax = plt.subplots(2, 1, figsize = (10, 20), sharex = True, sharey = True)
ax[0].contour(xs, ys, z, 0)
ax[0].scatter(x = positive_x1, y = positive_x2, marker = 'x', color = 'r')
ax[0].scatter(x = negative_x1, y = negative_x2, color = 'lightgreen')

ax[1].contour(xs, ys, z_2, 0)
ax[1].scatter(x = positive_x1, y = positive_x2, marker = 'x', color = 'r')
ax[1].scatter(x = negative_x1, y = negative_x2, color = 'lightgreen')

ax[0].set_title(f'lambda = {lambda_01}', fontsize = 30)
ax[1].set_title(f'lambda = {lambda_02}', fontsize = 30)

Text(0.5, 1.0, 'lambda = 10')

从结果来看，当$\lambda =0.1$时，拟合结果较好，当$\lambda =10$时，拟合结果具有明显的高偏差问题