根据以上两篇的分析,下面我们还要对数据进行处理,观察Age和Fare两个属性,乘客的数值变化幅度较大!根据逻辑回归和梯度下降的了解,如果属性值之间scale差距较大,将对收敛速度造成较大影响,甚至不收敛!因此,我们需要运用scikit-learn里面的preprocessing模块对Age和Fare两个属性做一个scaling,即将其数值转化为[-1,1]范围内。
1 # 接下来我们将一些变化幅度较大的特征化到[-1,1]之内,这样可以加速logistic regression的收敛 2 import sklearn.preprocessing as preprocessing 3 scaler = preprocessing.StandardScaler() 4 age_scale_param = scaler.fit(df['Age']) 5 df['Age_scaled'] = scaler.fit_transform(df['Age'],age_scale_param) 6 fare_scale_param = scaler.fit(df['Fare']) 7 df['Fare_scaled'] = scaler.fit_transform(df['Fare'],fare_scale_param) 8 print(df)
| PassengerId | Survived | Age | SibSp | Parch | Fare | Cabin_No | Cabin_Yes | Embarked_C | Embarked_Q | Embarked_S | Sex_female | Sex_male | Pclass_1 | Pclass_2 | Pclass_3 | Age_scaled | Fare_scaled |
0 | 1 | 0 | 22.000000 | 1 | 0 | 7.2500 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -0.561417 | -0.502445 |
1 | 2 | 1 | 38.000000 | 1 | 0 | 71.2833 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0.613177 | 0.786845 |
2 | 3 | 1 | 26.000000 | 0 | 0 | 7.9250 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | -0.267768 | -0.488854 |
3 | 4 | 1 | 35.000000 | 1 | 0 | 53.1000 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | 0.392941 | 0.420730 |
4 | 5 | 0 | 35.000000 | 0 | 0 | 8.0500 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 0.392941 | -0.486337 |
5 | 6 | 0 | 23.828953 | 0 | 0 | 8.4583 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | -0.427149 | -0.478116 |
6 | 7 | 0 | 54.000000 | 0 | 0 | 51.8625 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 1.787771 | 0.395814 |
7 | 8 | 0 | 2.000000 | 3 | 1 | 21.0750 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -2.029659 | -0.224083 |
8 | 9 | 1 | 27.000000 | 0 | 2 | 11.1333 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | -0.194356 | -0.424256 |
9 | 10 | 1 | 14.000000 | 1 | 0 | 30.0708 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | -1.148714 | -0.042956 |
10 | 11 | 1 | 4.000000 | 1 | 1 | 16.7000 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | -1.882835 | -0.312172 |
11 | 12 | 1 | 58.000000 | 0 | 0 | 26.5500 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | 2.081420 | -0.113846 |
12 | 13 | 0 | 20.000000 | 0 | 0 | 8.0500 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -0.708241 | -0.486337 |
13 | 14 | 0 | 39.000000 | 1 | 5 | 31.2750 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 0.686589 | -0.018709 |
14 | 15 | 0 | 14.000000 | 0 | 0 | 7.8542 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | -1.148714 | -0.490280 |
15 | 16 | 1 | 55.000000 | 0 | 0 | 16.0000 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1.861183 | -0.326267 |
16 | 17 | 0 | 2.000000 | 4 | 1 | 29.1250 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | -2.029659 | -0.061999 |
17 | 18 | 1 | 32.066493 | 0 | 0 | 13.0000 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0.177586 | -0.386671 |
18 | 19 | 0 | 31.000000 | 1 | 0 | 18.0000 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 0.099292 | -0.285997 |
19 | 20 | 1 | 29.518205 | 0 | 0 | 7.2250 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | -0.009489 | -0.502949 |
20 | 21 | 0 | 35.000000 | 0 | 0 | 26.0000 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0.392941 | -0.124920 |
21 | 22 | 1 | 34.000000 | 0 | 0 | 13.0000 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 0.319529 | -0.386671 |
22 | 23 | 1 | 15.000000 | 0 | 0 | 8.0292 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | -1.075302 | -0.486756 |
23 | 24 | 1 | 28.000000 | 0 | 0 | 35.5000 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | -0.120944 | 0.066360 |
24 | 25 | 0 | 8.000000 | 3 | 1 | 21.0750 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | -1.589186 | -0.224083 |
25 | 26 | 1 | 38.000000 | 1 | 5 | 31.3875 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 0.613177 | -0.016444 |
26 | 27 | 0 | 29.518205 | 0 | 0 | 7.2250 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | -0.009489 | -0.502949 |
27 | 28 | 0 | 19.000000 | 3 | 2 | 263.0000 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | -0.781653 | 4.647001 |
28 | 29 | 1 | 22.380113 | 0 | 0 | 7.8792 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | -0.533512 | -0.489776 |
29 | 30 | 0 | 27.947206 | 0 | 0 | 7.8958 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -0.124820 | -0.489442 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
861 | 862 | 0 | 21.000000 | 1 | 0 | 11.5000 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | -0.634829 | -0.416873 |
862 | 863 | 1 | 48.000000 | 0 | 0 | 25.9292 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | 1.347299 | -0.126345 |
863 | 864 | 0 | 10.888325 | 8 | 2 | 69.5500 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | -1.377148 | 0.751946 |
864 | 865 | 0 | 24.000000 | 0 | 0 | 13.0000 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | -0.414592 | -0.386671 |
865 | 866 | 1 | 42.000000 | 0 | 0 | 13.0000 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 0.906826 | -0.386671 |
866 | 867 | 1 | 27.000000 | 1 | 0 | 13.8583 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | -0.194356 | -0.369389 |
867 | 868 | 0 | 31.000000 | 0 | 0 | 50.4958 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0.099292 | 0.368295 |
868 | 869 | 0 | 25.977889 | 0 | 0 | 9.5000 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -0.269391 | -0.457142 |
869 | 870 | 1 | 4.000000 | 1 | 1 | 11.1333 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -1.882835 | -0.424256 |
870 | 871 | 0 | 26.000000 | 0 | 0 | 7.8958 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -0.267768 | -0.489442 |
871 | 872 | 1 | 47.000000 | 1 | 1 | 52.5542 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | 1.273886 | 0.409741 |
872 | 873 | 0 | 33.000000 | 0 | 0 | 5.0000 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0.246117 | -0.547748 |
873 | 874 | 0 | 47.000000 | 0 | 0 | 9.0000 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1.273886 | -0.467209 |
874 | 875 | 1 | 28.000000 | 1 | 0 | 24.0000 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | -0.120944 | -0.165189 |
875 | 876 | 1 | 15.000000 | 0 | 0 | 7.2250 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | -1.075302 | -0.502949 |
876 | 877 | 0 | 20.000000 | 0 | 0 | 9.8458 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -0.708241 | -0.450180 |
877 | 878 | 0 | 19.000000 | 0 | 0 | 7.8958 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -0.781653 | -0.489442 |
878 | 879 | 0 | 27.947206 | 0 | 0 | 7.8958 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -0.124820 | -0.489442 |
879 | 880 | 1 | 56.000000 | 0 | 1 | 83.1583 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1.934596 | 1.025945 |
880 | 881 | 1 | 25.000000 | 0 | 1 | 26.0000 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | -0.341180 | -0.124920 |
881 | 882 | 0 | 33.000000 | 0 | 0 | 7.8958 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 0.246117 | -0.489442 |
882 | 883 | 0 | 22.000000 | 0 | 0 | 10.5167 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | -0.561417 | -0.436671 |
883 | 884 | 0 | 28.000000 | 0 | 0 | 10.5000 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | -0.120944 | -0.437007 |
884 | 885 | 0 | 25.000000 | 0 | 0 | 7.0500 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | -0.341180 | -0.506472 |
885 | 886 | 0 | 39.000000 | 0 | 5 | 29.1250 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 0.686589 | -0.061999 |
886 | 887 | 0 | 27.000000 | 0 | 0 | 13.0000 | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | -0.194356 | -0.386671 |
887 | 888 | 1 | 19.000000 | 0 | 0 | 30.0000 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | -0.781653 | -0.044381 |
888 | 889 | 0 | 16.232379 | 1 | 2 | 23.4500 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | -0.984830 | -0.176263 |
889 | 890 | 1 | 26.000000 | 0 | 0 | 30.0000 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | -0.267768 | -0.044381 |
890 | 891 | 0 | 32.000000 | 0 | 0 | 7.7500 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0.172705 | -0.492378 |
891 rows × 18 columns
接下来我们把需要的feature字段取出来,转成numpy格式,使用scikit-learn中的LogisticRegression建模。
1 from sklearn import linear_model 2 train_df = df.filter(regex='Survived|Age_.*|SibSp|Parch|Fare_.*|Cabin_.*|Embarked_.*|Sex_.*|Pclass_.*') 3 train_np = train_df.as_matrix() 4 y = train_np[:,0] 5 X = train_np[:,1:] 6 clf = linear_model.LogisticRegression(C=1.0,penalty='l1',tol=1e-6) 7 clf.fit(X,y) 8 print(clf)
接下来我们需要对测试数据集和训练数据集做一样的操作
1 # #首先用同样的RandomForestRegressor模型填上丢失的年龄 2 data_test = pd.read_csv("test.csv") 3 data_test.loc[ (data_test.Fare.isnull()), 'Fare' ] = 0 4 5 tmp_df = data_test[['Age','Fare', 'Parch', 'SibSp', 'Pclass']] 6 null_age = tmp_df[data_test.Age.isnull()].as_matrix() 7 # 根据特征属性X预测年龄并补上 8 X = null_age[:, 1:] 9 predictedAges = rfr.predict(X) 10 data_test.loc[ (data_test.Age.isnull()), 'Age' ] = predictedAges 11 12 data_test = set_Cabin_type(data_test) 13 dummies_Cabin = pd.get_dummies(data_test['Cabin'], prefix= 'Cabin') 14 dummies_Embarked = pd.get_dummies(data_test['Embarked'], prefix= 'Embarked') 15 dummies_Sex = pd.get_dummies(data_test['Sex'], prefix= 'Sex') 16 dummies_Pclass = pd.get_dummies(data_test['Pclass'], prefix= 'Pclass') 17 18 19 df_test = pd.concat([data_test, dummies_Cabin, dummies_Embarked, dummies_Sex, dummies_Pclass], axis=1) 20 df_test.drop(['Pclass', 'Name', 'Sex', 'Ticket', 'Cabin', 'Embarked'], axis=1, inplace=True) 21 df_test['Age_scaled'] = scaler.fit_transform(df_test['Age'], age_scale_param) 22 df_test['Fare_scaled'] = scaler.fit_transform(df_test['Fare'], fare_scale_param) 23 24 test = df_test.filter(regex='Age_.*|SibSp|Parch|Fare_.*|Cabin_.*|Embarked_.*|Sex_.*|Pclass_.*') 25 predictions = clf.predict(test) 26 result = pd.DataFrame({'PassengerId':data_test['PassengerId'].as_matrix(), 'Survived':predictions.astype(np.int32)}) 27 result.to_csv("logistic_regression_predictions.csv", index=False)
最后我们将预测的结果保存在logistic_regression_predictions.csv文件里。到这里只是简单分析过后的一个baseline系统。
下面要判定一下当前模型所处状态(欠拟合或者过拟合)
这里面有个问题,前面不断地做特征工程,产生的特征越来越多,用这些特征来训练模型,会对训练集拟合得越来越好,同时也可能在逐步丧失泛化能力,从而在待测数据上表现不佳,也就是发生过拟合问题。
从另一个角度上说,如果模型在待测数据上表现不佳,除掉上述说的过拟合问题,有时候也存在欠拟合问题,也就是说在训练集上拟合的结果也不好。
对于过拟合和欠拟合,有两种优化方式:
1.过拟合
1)做一个feature selection,选较好的feature的subset来做training
2)提供更多的数据,从而弥补原始数据偏差,从而学习到的模型更准确
3)采用正则化,正则化方法包括L0正则、L1正则和L2正则,而正则一般是在目标函数之后加上对于的范数。但是在机器学习中一般使用L2正则。
2.欠拟合
1)添加其他特征项,有时候我们模型出现欠拟合的时候是因为特征项不够导致的,可以添加其他特征项来很好地解决。例如,“组合”、“泛化”、“相关性”三类特征是特征添加的重要手段。
2)添加多项式特征,这个在机器学习算法里面用的很普遍,例如将线性模型通过添加二次项或者三次项使模型泛化能力更强。
3)减少正则化参数,正则化的目的是用来防止过拟合的,但是现在模型出现了欠拟合,则需要减少正则化参数。
scikit-learn里面的learning curve可以帮助我们判定模型现在所处的状态。这里我们画一下最先得到的baseline model的learning curve。
1 import numpy as np 2 import matplotlib.pyplot as plt 3 from sklearn.learning_curve import learning_curve 4 5 #用sklearn的learning_curve得到training_score和cv_score,使用matplotlib画出learning curve 6 7 def plot_learning_curve(estimator,title,X,y,ylim=None,cv=None,n_jobs=1,train_sizes=np.linspace(.05,1.,20),verbose=0,plot=True): 8 """ 9 画出data在某模型上的learning curve. 10 参数解释 11 ---------- 12 estimator : 你用的分类器。 13 title : 表格的标题。 14 X : 输入的feature,numpy类型 15 y : 输入的target vector 16 ylim : tuple格式的(ymin, ymax), 设定图像中纵坐标的最低点和最高点 17 cv : 做cross-validation的时候,数据分成的份数,其中一份作为cv集,其余n-1份作为training(默认为3份) 18 n_jobs : 并行的的任务数(默认1) 19 """ 20 train_sizes, train_scores, test_scores = learning_curve( 21 estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes, verbose=verbose) 22 23 train_scores_mean = np.mean(train_scores, axis=1) 24 train_scores_std = np.std(train_scores, axis=1) 25 test_scores_mean = np.mean(test_scores, axis=1) 26 test_scores_std = np.std(test_scores, axis=1) 27 28 if plot: 29 plt.figure() 30 plt.title(title) 31 if ylim is not None: 32 plt.ylim(*ylim) 33 plt.xlabel('train_sample') 34 plt.ylabel('score') 35 plt.gca().invert_yaxis() 36 plt.grid() 37 38 plt.fill_between(train_sizes, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, 39 alpha=0.1, color="b") 40 plt.fill_between(train_sizes, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, 41 alpha=0.1, color="r") 42 plt.plot(train_sizes, train_scores_mean, 'o-', color="b", label="train_score") 43 plt.plot(train_sizes, test_scores_mean, 'o-', color="r", label="cross_validation_score") 44 45 plt.legend(loc='best') 46 plt.draw() 47 plt.gca().invert_yaxis() 48 plt.show() 49 50 midpoint = ((train_scores_mean[-1] + train_scores_std[-1]) + (test_scores_mean[-1] - test_scores_std[-1])) / 2 51 diff = (train_scores_mean[-1] + train_scores_std[-1]) - (test_scores_mean[-1] - test_scores_std[-1]) 52 return midpoint, diff 53 54 plot_learning_curve(clf, u"学习曲线", X, y)
结果如下(0.80656968448540245, 0.018258876711338634)
从得到的结果来看,尽管learn——curve没有理论推导的光滑,但是仍可以看出,训练集和交叉验证集上的得分曲线走势还是符合预期的。
目前的曲线来看,我们的model并不处于overfitting的状态(overfitting的表现一般是训练集得分高,交叉验证集得分低很多,中间的gap比较大)。因此我们可以再做些feature engineering的工作,添加一些新的特征或者组合特征到模型中。