Feature Engineering Made Easy 笔记 Chapter5

Chapter 5 Feature Selection

预测性能指标

1. 真阳性率和假阳性率
2. 绝对平均误差(回归)
3. $R^2=1-\frac{S{S}_{res}}{S{S}_{total}}$ coefficient of determination
   $S{S}_{res}=\sum (y_i-\hat{y}{)}^2$
   $S{S}_{total}=\sum (y_i-\overline{y}{)}^2$
   模型越好，越接近1
4. 训练时间
5. 预测新数据时间
6. …

基于统计的特征选择

1. Pearson Correlation $\frac{Cov(x,y)}{{\sigma }_x{\sigma }_y}$ 衡量两个变量的线性关系。
2. Hypothesis Test:
   Anova F value
   chi-squared
   使用SelectKBest

k_best = SelectKBest(f_classif, k=5)
k_best.fit_transform(X, y)

基于机器学习模型选择

使用机器学习 SelectFromModel
Decision Tree
logisitic regression（针对回归任务）
SVC（针对二分数据集）

tree = DecisionTreeClassifier()
tree.fit(X, y)
tree_pipe_params = {'classifier__max_depth': [1, 3, 5, 7]}
importances = pd.DataFrame({'importance': tree.feature_importances_, 'feature':X.columns}).sort_values('importance', ascending=False)
tree_pipe_params = {'classifier__max_depth': [1, 3, 5, 7]}
select_from_model = SelectFromModel(DecisionTreeClassifier(), 
                                    threshold=.05)
selected_X = select_from_model.fit_transform(X, y)                                   

from sklearn.pipeline import Pipeline
select_from_pipe = Pipeline([('select', SelectFromModel(DecisionTreeClassifier())), 
                             ('classifier', d_tree)])
select_from_pipe_params = deepcopy(tree_pipe_params)
select_from_pipe_params.update({
              'select__threshold': [.01, .05, .1, "mean", "median", "2.*mean"],
              'select__estimator__max_depth': [None, 1, 3, 5, 7]
              })
print select_from_pipe_params
# not better than original
get_best_model_and_accuracy(select_from_pipe, 
                            select_from_pipe_params, 
                            X, y)