Read#Feature selection using Wrapper methods in Python

Feature selection using Wrapper methods in Python

To perform any machine learning task or to get insights from such high dimensional data

Feature selection:

• Training faster
• Reduce complexity
• Better prediction
• Reduce overfitting

Feature selection methods:

• Filter method
• Wrapper method
• Embedded method
  

Wrapper method

（based on specific machine learning algorithm）
It follows a greedy search approach by evaluating all the possible combination of feature against evaluation criterion. Finally, it selects the combination of features that gives the optimal results for the specified ML algorithm.

Most common method in wrapper :
1.Forward selection
2.Backward elimination
3.BI-directional elimination (Stepwise Selection)

in sklearn

1.Forward selection
start with a null model and then fit the model with each individual one at a time and select the feature with the minimum p-value.

In short, the steps for forward selection technique are as follows :
1.Choose a significance level.（显著性水平）
2.Fit model one feature at a time. Select the feature with the lowest p-value.
3.Fit all possible models with one extra feature.
4.Again,select the feature with minimum p-value.if p-value < significance level then go to step 3 , otherwise terminate the process.

from sklearn.datasets import load_bosto
import pandas as pd
import statsmodels.api as sm
boston = load_boston()
bos = pd.DataFrame(boston.data, columns=boston.feature_names)
bos['Price'] = boston.target
X = bos.drop("Price", 1)
y = bos['Price']
bos.head()
def forward_selection(data, target, significance_level=0.05):
    initial_features = data.columns.tolist()
    best_features = []
    while (len(list(initial_features))>0):
        remaining_features = list(set(initial_features)-set(best_features))
        new_pval = pd.Series(index=remaining_features)
        for new_column in remaining_features:
            model = sm.OLS(target, sm.add_constant(data[best_features+[new_column]])).fit()
            new_pval[new_column] = model.pvalues[new_column]
        min_p_value = new_pval.min()
        if(min_p_value<significance_level):
            best_features.append(new_pval.idxmin())
        else:
            break
    return best_features

print(forward_selection(X,y))

2. Backward elimination
In short, the steps for backward elimination technique are as follows :
1.Choose a significance level (e.g. SL = 0.05 with a 95% confidence).
2.Fit a full model including all the features.
3.Consider the feature with highest p-value. If the p-value > significance level then go to Step 4, otherwise terminate the process.
4.Remove the feature which is under consideration.
5.Fit a model without this feature. Repeat the entire process from Step 3.