Spark之特征选择

最新推荐文章于 2024-08-16 21:23:43 发布

mosu027

最新推荐文章于 2024-08-16 21:23:43 发布

阅读量1.6k

点赞数

分类专栏：机器学习文章标签： spark 机器学习

本文链接：https://blog.csdn.net/mosu027/article/details/62430424

版权

机器学习专栏收录该内容

3 篇文章 0 订阅

订阅专栏

特征选择

坊间传言：数据和特征决定了机器学习的上限,而模型和算法只是逼近这个上限而已。因此特征工程显得尤为重要。最近的工作中做了较多的特征工作，在这里做一个小小的总结。

peason特征选择

笔者本身统计学的，一直认为pearson也只能做做相关性分析，判断两个变量相关性什么的。在工作中刚开始并未觉得卡方特征选择效果会有多好，于是退而求其次，选择了peason,这个是spark里特有的。废话不多说，直接上代码吧，公式你们都会懂的，就不贴了。

for(String featurename:featurenames){
    cols.add(functions.corr("label", featurename).alias(featurename));
}
Dataset<Row> featruecorr = feature.
        agg(functions.first("label"), (scala.collection.Seq)scala.collection.JavaConversions.asScalaBuffer(cols));

Row[] tempvalueRows = (Row[]) featruecorr.collect();
String tempfeaturenames[] = featruecorr.columns();

ArrayList<String> tempArrayList = new ArrayList<String>();
ArrayList<String> tempcorrList = new ArrayList<String>();

for(int i = 1; i < tempvalueRows[0].length(); i++){
    if (tempvalueRows[0].get(i) instanceof Double){
        if(Math.abs((Double)tempvalueRows[0].get(i)) > threshhold){
            tempArrayList.add(tempfeaturenames[i]);
            tempcorrList.add(String.valueOf(tempvalueRows[0].get(i)));
        }
    }
}

String[]  newfeaturenames =  new String[tempArrayList.size()];
newfeaturenames = (String[])tempArrayList.toArray(newfeaturenames);
System.out.println("newfeaturenames: ");
for(int i =0;i < newfeaturenames.length;i++){
    System.out.println(String.valueOf(newfeaturenames[i]) + "  " + tempcorrList.get(i));
}
System.out.println("rawfeaturenames num is " + featurenames.length);
System.out.println("newfeaturenames num is " + newfeaturenames.length);

缺点：pearson是假设两组数据是连续变量，而我们通常处理分类问题时，label列是离散的，无法解决非线性相关问题。

卡方特征选择

卡方特征选择是基于卡方检验的思想（理解了卡方布，更有助于如何设置参数），其效果也是相对较好的。

public static String[] feautureSelectionByChiSq(Dataset<Row> feature, Double    threshholdfpr,Integer featurenum, String[] featurenames){

        VectorAssembler va = new VectorAssembler()
            .setInputCols(featurenames).setOutputCol("features");
        feature =  va.transform(feature);
        Dataset<Row> newfeature = feature.select("userid", "features", "label");
        feature.unpersist();
        // scaler
        StandardScaler scaler = new StandardScaler()
            .setInputCol("features")
            .setOutputCol("scaledFeatures")
            .setWithStd(true)
            .setWithMean(false);
        Dataset<Row> scalerfeature = scaler.fit(newfeature).transform(newfeature);
        // 
        ChiSqSelector selector = null;
        if(featurenum < 0){
            selector = new ChiSqSelector()
              .setFpr(threshholdfpr)
//            .setPercentile(threshholdfpr)
              .setFeaturesCol("scaledFeatures")
              .setLabelCol("label")
              .setOutputCol("selectedFeatures");
        }else{
            selector = new ChiSqSelector()
              .setNumTopFeatures(featurenum)
              .setFeaturesCol("scaledFeatures")
              .setLabelCol("label")
              .setOutputCol("selectedFeatures");
        }
        ChiSqSelectorModel selectorModel = selector.fit(scalerfeature);

        ArrayList<String> tempList = new ArrayList<String>();
        System.out.println(selectorModel.selectedFeatures());
        for(int i = 0; i < selectorModel.selectedFeatures().length; i++ ){
            tempList.add(featurenames[i]);
        }

        System.out.println(selectorModel.getFpr());
        String[]  newfeaturenames =  new String[tempList.size()];
        newfeaturenames = (String[])tempList.toArray(newfeaturenames);

        return newfeaturenames;
    }

随机森林特征选择

long long ago就看到了可以用rf进行特征选择，可在spark里一直苦苦不得其法，最近刚好在一大牛的博客看到，豁然开朗，先放代码，原理下次再放，今天比较懒。

    VectorAssembler va = new VectorAssembler()
        .setInputCols(featurenames).setOutputCol("features");
    feature =  va.transform(feature);
    Dataset<Row> newfeature = feature.select("userid", "features", "label");
    feature.unpersist();
    // scaler
    StandardScaler scaler = new StandardScaler()
        .setInputCol("features")
        .setOutputCol("scaledFeatures")
        .setWithStd(true)
        .setWithMean(false);
    Dataset<Row> scalerfeature = scaler.fit(newfeature).transform(newfeature);
    // 
    RandomForestClassifier rf = new RandomForestClassifier()
              .setLabelCol("label")
              .setFeaturesCol("scaledFeatures");
    RandomForestClassificationModel model = rf.fit(scalerfeature);
    HashMap<String, Double> featurenameMap = new HashMap<String, Double>();

    ArrayList<String> tempList = new ArrayList<String>();

    for(int i =0; i < featurenames.length; i++){
        featurenameMap.put(featurenames[i], model.featureImportances().toArray()[i]);
        if(model.featureImportances().toArray()[i] > threshhold){
            tempList.add(featurenames[i]);
        }
    }
    String[]  newfeaturenames =  new String[tempList.size()];
    newfeaturenames = (String[])tempList.toArray(newfeaturenames);