sklearn.feature_selection模块中的类可用于样本集的特征选择/降维,以提高估计器的准确性得分或提高其在非常高维的数据集上的性能。
移除低方差特征:
VarianceThreshold是一种简单的特征选择基线方法。它删除方差不满足某个阈值的所有特征。默认情况下,它会删除所有零方差特征,即在所有样本中具有相同值的特征。注解:其实需要预先判断数据分布,计算方差阈值
示例:
例如,假设我们有一个具有布尔特征的数据集,并且我们希望在80%以上的样本中删除所有为1或为0(开或关)的特征。布尔特征是伯努利随机变量,这类变量的方差由
V
a
r
[
X
]
=
p
(
1
−
p
)
Var[X]=p(1-p)
Var[X]=p(1−p)
所以我们可以选择使用阈值
0.8
∗
(
1
−
0.8
)
0.8* (1 - 0.8)
0.8∗(1−0.8):
#代码:
from sklearn.feature_selection import VarianceThreshold
feature = [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 1, 1], [0, 1, 0], [0, 1, 1]]
sel = VarianceThreshold(threshold=(.8 * (1 - .8)))
feature_1 = sel.fit_transform(feature)
print(feature_1)
#[[0 1], [1 0], [0 0],[1 1],[1 0],[1 1]]
单变量特征选择
单变量特征选择的工作原理是根据单变量统计检验选择最佳特征。它可以被看作是一个预处理步骤估计。Scikit-learn将特性选择例程公开为实现转换方法的对象:
1、SelectKBest删除除k个得分最高的特征以外的所有特征
2、SelectPercentile删除除用户指定的得分百分比最高的特征外的所有特征
对每个特征使用常见的单变量统计检验:假阳性率SelectFpr、假发现率SelectFdr或簇错误SelectFwe。
3、GenericUnivariateSelect允许使用可配置策略执行单变量特征选择。这允许选择最佳的单变量选择策略与超参数搜索估计。
例如,我们可以对样本进行χ2检验,仅检索两个最佳特征,如下所示
from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2
x_data, y = load_iris(return_X_y=True)
x_data_selected = SelectKBest(chi2, k=2).fit_transform(x_data, y)
这些对象以一个评分函数作为输入,该函数返回单变量分数和p值(或仅用于SelectKBest和SelectPercentile):注意不能混着用。同样可以用于稀疏数据,返回的也是稀疏数据:Feature selection with sparse data
1、回归
f_regression, mutual_info_regression
2、分类
chi2, f_classif, mutual_info_classif
示例:
使用单变量的特征选择,以提高分类精度的噪声数据集。
在本例中,一些嘈杂(非信息性)特征被添加到iris数据集。采用支持向量机(SVM)对单变量特征选择前后的数据集进行分类。对于每个特征,我们绘制单变量特征选择的p值和相应的SVM权重。有了这个,我们将比较模型的准确性,并研究单变量特征选择模型权重的影响。
# Generate sample data
import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
# The iris dataset
X, y = load_iris(return_X_y=True)
# 添加噪声数据
E = np.random.RandomState(42).uniform(0, 0.1, size=(X.shape[0], 20))
X = np.hstack((X, E))
# 划分数据集
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)
# 特征选择
from sklearn.feature_selection import SelectKBest, f_classif
selector = SelectKBest(f_classif, k=4)
selector.fit(X_train, y_train)
scores = -np.log10(selector.pvalues_)
scores /= scores.max()
# %%
import matplotlib.pyplot as plt
X_indices = np.arange(X.shape[-1])
plt.figure(1)
plt.clf()
plt.bar(X_indices - 0.05, scores, width=0.2)
plt.title("Feature univariate score")
plt.xlabel("Feature number")
plt.ylabel(r"Univariate score ($-Log(p_{value})$)")
plt.show()
#没有选择数据
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import LinearSVC
clf = make_pipeline(MinMaxScaler(), LinearSVC())
clf.fit(X_train, y_train)
print(
"Classification accuracy without selecting features: {:.3f}".format(
clf.score(X_test, y_test)
)
)
svm_weights = np.abs(clf[-1].coef_).sum(axis=0)
svm_weights /= svm_weights.sum()
# 淑君选择后
clf_selected = make_pipeline(SelectKBest(f_classif, k=4), MinMaxScaler(), LinearSVC())
clf_selected.fit(X_train, y_train)
print(
"Classification accuracy after univariate feature selection: {:.3f}".format(
clf_selected.score(X_test, y_test)
)
)
svm_weights_selected = np.abs(clf_selected[-1].coef_).sum(axis=0)
svm_weights_selected /= svm_weights_selected.sum()
plt.bar(
X_indices - 0.45, scores, width=0.2, label=r"Univariate score ($-Log(p_{value})$)"
)
plt.bar(X_indices - 0.25, svm_weights, width=0.2, label="SVM weight")
plt.bar(
X_indices[selector.get_support()] - 0.05,
svm_weights_selected,
width=0.2,
label="SVM weights after selection",
)
plt.title("Comparing feature selection")
plt.xlabel("Feature number")
plt.yticks(())
plt.axis("tight")
plt.legend(loc="upper right")
plt.show()
"""
结果分析:使用特征选择后,SVM准确率从0.79->0.87。而且从图中可以看到SVM的权重更关注原来的4个重要的特征,而不关注噪声。
"""
示例:
Comparison of F-test and mutual information
这个例子说明了单变量F检验统计量和互信息之间的区别。
我们考虑均匀分布在[0,1]上的3个特征
x
1
,
x
2
,
x
3
x_1,x_2,x_3
x1,x2,x3,目标函数依赖于它们如下:
y
=
x
1
+
s
i
n
(
6
∗
p
i
∗
x
2
)
+
0.1
∗
N
(
0
,
1
)
y = x_1 + sin(6 * pi * x_2) + 0.1 * N(0, 1)
y=x1+sin(6∗pi∗x2)+0.1∗N(0,1)
即第三个特征是完全不相关的。
下面的代码绘制了
y
y
y对单个
x
i
x_i
xi的依赖关系以及单变量F检验统计量和互信息的归一化值。
import numpy as np
import matplotlib.pyplot as plt
from sklearn.feature_selection import f_regression, mutual_info_regression
np.random.seed(0)
X = np.random.rand(1000, 3)
y = X[:, 0] + np.sin(6 * np.pi * X[:, 1]) + 0.1 * np.random.randn(1000)
f_test, _ = f_regression(X, y)
f_test /= np.max(f_test)
mi = mutual_info_regression(X, y)
mi /= np.max(mi)
plt.figure(figsize=(15, 5))
for i in range(3):
plt.subplot(1, 3, i + 1)
plt.scatter(X[:, i], y, edgecolor="black", s=20)
plt.xlabel("$x_{}$".format(i + 1), fontsize=14)
if i == 0:
plt.ylabel("$y$", fontsize=14)
plt.title("F-test={:.2f}, MI={:.2f}".format(f_test[i], mi[i]), fontsize=16)
plt.show()
分析:由于 F − t e s t F-test F−test只捕获了线性相关性,它将 x 1 x_1 x1列为最具鉴别力的特征。另一方面,互信息可以捕捉变量之间的任何类型的依赖关系,它将 x 2 x_2 x2评为最有鉴别力的特征,这可能更符合我们对这个例子的直觉感知。这两种方法都正确地将 x 3 x_3 x3标记为不相关。
递归特征消除
给定一个外部的估计,分配权重的特征(例如,一个线性模型的系数),递归特征消除(RFE)的目标是选择特征递归考虑越来越小的特征集。首先,估计器在初始特征集上训练,每个特征的重要性通过任何特定属性(如coef_,feature_importance_)或可调用获得。然后,从当前的特征集合中删除最不重要的特征。该过程在修剪集上递归地重复,直到最终达到所需选择的特征数。
RFECV在交叉验证循环中执行RFE,以找到最佳的特征数量。
示例:
from sklearn.svm import SVC
from sklearn.datasets import load_digits
from sklearn.feature_selection import RFE
import matplotlib.pyplot as plt
# Load the digits dataset
digits = load_digits()
X = digits.images.reshape((len(digits.images), -1))
y = digits.target
# Create the RFE object and rank each pixel
svc = SVC(kernel="linear", C=1)
rfe = RFE(estimator=svc, n_features_to_select=1, step=1)
rfe.fit(X, y)
ranking = rfe.ranking_.reshape(digits.images[0].shape)
# Plot pixel ranking
plt.matshow(ranking, cmap=plt.cm.Blues)
plt.colorbar()
plt.title("Ranking of pixels with RFE")
plt.show()
结合网格搜索的pipline的示例:
import matplotlib.pyplot as plt
from sklearn.svm import SVC
from sklearn.model_selection import StratifiedKFold
from sklearn.feature_selection import RFECV
from sklearn.datasets import make_classification
# Build a classification task using 3 informative features
X, y = make_classification(
n_samples=1000,
n_features=25,
n_informative=3,
n_redundant=2,
n_repeated=0,
n_classes=8,
n_clusters_per_class=1,
random_state=0,
)
# Create the RFE object and compute a cross-validated score.
svc = SVC(kernel="linear")
# The "accuracy" scoring shows the proportion of correct classifications
min_features_to_select = 1 # Minimum number of features to consider
rfecv = RFECV(
estimator=svc,
step=1,
cv=StratifiedKFold(2),
scoring="accuracy",
min_features_to_select=min_features_to_select,
)
rfecv.fit(X, y)
print("Optimal number of features : %d" % rfecv.n_features_)
# Plot number of features VS. cross-validation scores
plt.figure()
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score (accuracy)")
plt.plot(
range(min_features_to_select, len(rfecv.grid_scores_) + min_features_to_select),
rfecv.grid_scores_,
)
plt.show()
基于模型的SelectFromModel
1、L1范数惩罚的线性模型
受L1范数惩罚的线性模型具有稀疏的解:它们的许多估计系数为零。当目标是降低数据的维数以与另一个分类器一起使用时,它们可以与SelectFromModel一起使用以选择非零系数。特别是,用于此目的的稀疏估计是Lasso用于回归,LogisticRegression和LinearSVC用于分类:
from sklearn.svm import LinearSVC
from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectFromModel
X, y = load_iris(return_X_y=True)
X.shape
lsvc = LinearSVC(C=0.01, penalty="l1", dual=False).fit(X, y)
model = SelectFromModel(lsvc, prefit=True)
X_new = model.transform(X)
X_new.shape
L1和压缩感知
对于一个很好的alpha选择,Lasso可以完全恢复非零变量的精确集合,只需使用很少的观测,前提是满足某些特定条件。特别是,样本的数量应该“足够大”,或者L1模型将随机执行,其中“足够大”取决于非零系数的数目、特征数的对数、噪声量、非零参数的最小绝对值以及设计矩阵X的结构。此外,设计矩阵必须显示某些特定属性,例如相关性不太高。
对于非零系数的恢复,没有选择alpha参数的一般规则。它可以通过交叉验证(LassoCV或LassoLarsCV)来设置,尽管这可能会导致惩罚不足的模型:包括少量的非相关变量对预测得分是无害的。BIC(LassoLarsIC)则倾向于设置较高的alpha值。
2、基于树的
基于树的估计器(请参见sklearn.tree模块和sklear.ensemble模块中的森林)可用于计算基于杂质的特征重要性,而这反过来又可用于丢弃不相关的特征(与SelectFromModel元转换器结合使用时)
import matplotlib.pyplot as plt
# 生成3个带有信息的特征的合成数据集
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
X, y = make_classification(
n_samples=1000,
n_features=10,
n_informative=3,
n_redundant=0,
n_repeated=0,
n_classes=2,
random_state=0,
shuffle=False,
)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=42)
# 随机森林分类器去拟合数据,计算特征重要性;
from sklearn.ensemble import RandomForestClassifier
feature_names = [f"feature {i}" for i in range(X.shape[1])]
forest = RandomForestClassifier(random_state=0)
forest.fit(X_train, y_train)
# feature_importances_属性并且计算在每一棵树的不纯度减少的均值和方差。
#基于不纯特性的重要性可能会误导high cardinality特征(许多独特的值)。可以使用'permutation_importance'。
import time
import numpy as np
start_time = time.time()
importances = forest.feature_importances_
std = np.std([tree.feature_importances_ for tree in forest.estimators_], axis=0)
elapsed_time = time.time() - start_time
print(f"Elapsed time to compute the importances: {elapsed_time:.3f} seconds")
# %%
# Let's plot the impurity-based importance.
import pandas as pd
forest_importances = pd.Series(importances, index=feature_names)
fig, ax = plt.subplots()
forest_importances.plot.bar(yerr=std, ax=ax)
ax.set_title("Feature importances using MDI")
ax.set_ylabel("Mean decrease in impurity")
fig.tight_layout()
# 可以看到,Permutation feature克服impurity-based feature importance的限制:没有对于high-cardinality features的偏差,并且可以计算留出的测试集。
from sklearn.inspection import permutation_importance
start_time = time.time()
result = permutation_importance(
forest, X_test, y_test, n_repeats=10, random_state=42, n_jobs=2
)
elapsed_time = time.time() - start_time
print(f"Elapsed time to compute the importances: {elapsed_time:.3f} seconds")
forest_importances = pd.Series(result.importances_mean, index=feature_names)
# 计算全部的permutation importance很费时,特征被混洗n次,并且模型重新训练去估计其重要性。画出重要性排序:importance ranking.
fig, ax = plt.subplots()
forest_importances.plot.bar(yerr=result.importances_std, ax=ax)
ax.set_title("Feature importances using permutation on full model")
ax.set_ylabel("Mean accuracy decrease")
fig.tight_layout()
plt.show()
#两种方法的效果是相同的特征被检测到,虽然重要的程度不同,MDI更不可能比permutation importance去完全忽略一个特征。
示例:对于人脸数据集进行并行的随机森林进行特征筛选:
#这个例子展示了如何使用树林来评估杂质,基于人脸图像数据集分类任务中像素的重要性,像素越热,越重要。
#下面的代码演示了如何可以在多个作业中并行化构造和预测。
# 首先,加载olivetti faces dataset和限制数据集前面5类,然后训练随机森林并且评估impurity-based feature importance,有一个缺点是不能评估分离的测试集。对于这个例子,我们感兴趣的是表示从完整的数据集学习的信息。此外,我们将设置用于任务的内核数。
from sklearn.datasets import fetch_olivetti_faces
n_jobs = -1 # 全部使用核心数,线程数
#加载数据集
data = fetch_olivetti_faces()
X, y = data.data, data.target
# 限制5类
mask = y < 5
X = X[mask]
y = y[mask]
# 随机森林分类器可以计算feature importances.
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators=750, n_jobs=n_jobs, random_state=42)
forest.fit(X, y)
# %%
# Feature importance based on mean decrease in impurity (MDI)
# feature_importances_属性并且计算在每一棵树的不纯度减少的均值和方差。
#基于不纯特性的重要性可能会误导high cardinality特征(许多独特的值)。可以使用'permutation_importance'。
import time
import matplotlib.pyplot as plt
start_time = time.time()
img_shape = data.images[0].shape
importances = forest.feature_importances_
elapsed_time = time.time() - start_time
print(f"Elapsed time to compute the importances: {elapsed_time:.3f} seconds")
imp_reshaped = importances.reshape(img_shape)
plt.matshow(imp_reshaped, cmap=plt.cm.hot)
plt.title("Pixel importances using impurity values")
plt.colorbar()
plt.show()
# 有限制的MDI对于这个数据集来说不是一个问题,因为:
# 1、所有的特征是有序的数字和并且没有基数偏差cardinality bias;
# 2、只对森林在训练集获得的表示知识感兴趣;
#如果不满足上面两个条件,需要用sklearn.inspection.permutation_importance`
顺序特征选择
另一种选择特征的方法是使用SequentialFeatureSelector(SFS)。SFS是一个贪婪的过程,在每次迭代时,我们选择最佳的新特征添加到我们所选择的特征基于交叉验证得分。也就是说,我们从0个特征开始,选择得分最高的最佳单特征。这个过程是重复的,直到我们达到所需数量的选定特征。我们也可以反方向(向后SFS),即从所有的特征开始,贪婪地选择要一个一个删除的特征。我们在这里说明这两种方法。
有趣的是,向前和向后选择选择了相同的特征集。一般来说,情况并非如此,两种方法会导致不同的结果。
我们还注意到,SFS选择的特征不同于那些选择的特征重要性:SFS选择BMI,而不是s1。虽然这听起来很合理,因为根据系数,BMI对应于第三个最重要的特征。考虑到SFS完全没有使用系数,这是相当了不起的。
最后,我们应该注意到SelectFromModel比SFS快得多。事实上,SelectFromModel只需要拟合一个模型一次,而SFS需要为每个迭代交叉验证许多不同的模型。然而,SFS适用于任何模型,而SelectFromModel要求底层估计器公开coef_attribute或feature_importance_attribute。向前SFS比向后SFS快,因为它只需要执行n_features_to_select=2次迭代,而向后的SFS需要执行N_特征n_features_to_select=8次迭代次数
from sklearn.datasets import load_diabetes
diabetes = load_diabetes()
X, y = diabetes.data, diabetes.target
print(diabetes.DESCR)
# %%
# Feature importance from coefficients
# ------------------------------------
#
# To get an idea of the importance of the features, we are going to use the
# :class:`~sklearn.linear_model.RidgeCV` estimator. The features with the
# highest absolute `coef_` value are considered the most important.
# We can observe the coefficients directly without needing to scale them (or
# scale the data) because from the description above, we know that the features
# were already standardized.
# For a more complete example on the interpretations of the coefficients of
# linear models, you may refer to
# :ref:`sphx_glr_auto_examples_inspection_plot_linear_model_coefficient_interpretation.py`.
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import RidgeCV
ridge = RidgeCV(alphas=np.logspace(-6, 6, num=5)).fit(X, y)
importance = np.abs(ridge.coef_)
feature_names = np.array(diabetes.feature_names)
plt.bar(height=importance, x=feature_names)
plt.title("Feature importances via coefficients")
plt.show()
# %%
# Selecting features based on importance
# --------------------------------------
#
# Now we want to select the two features which are the most important according
# to the coefficients. The :class:`~sklearn.feature_selection.SelectFromModel`
# is meant just for that. :class:`~sklearn.feature_selection.SelectFromModel`
# accepts a `threshold` parameter and will select the features whose importance
# (defined by the coefficients) are above this threshold.
#
# Since we want to select only 2 features, we will set this threshold slightly
# above the coefficient of third most important feature.
from sklearn.feature_selection import SelectFromModel
from time import time
threshold = np.sort(importance)[-3] + 0.01
tic = time()
sfm = SelectFromModel(ridge, threshold=threshold).fit(X, y)
toc = time()
print(f"Features selected by SelectFromModel: {feature_names[sfm.get_support()]}")
print(f"Done in {toc - tic:.3f}s")
# Selecting features with Sequential Feature Selection
# Another way of selecting features is to use
from sklearn.feature_selection import SequentialFeatureSelector
tic_fwd = time()
sfs_forward = SequentialFeatureSelector(
ridge, n_features_to_select=2, direction="forward"
).fit(X, y)
toc_fwd = time()
tic_bwd = time()
sfs_backward = SequentialFeatureSelector(
ridge, n_features_to_select=2, direction="backward"
).fit(X, y)
toc_bwd = time()
print(
"Features selected by forward sequential selection: "
f"{feature_names[sfs_forward.get_support()]}"
)
print(f"Done in {toc_fwd - tic_fwd:.3f}s")
print(
"Features selected by backward sequential selection: "
f"{feature_names[sfs_backward.get_support()]}"
)
print(f"Done in {toc_bwd - tic_bwd:.3f}s")
特征选择作为pipline的一部分
clf = Pipeline([
('feature_selection', SelectFromModel(LinearSVC(penalty="l1"))),
('classification', RandomForestClassifier())
])
clf.fit(X, y)