数据处理:
Dataset<Row> dataset = datasets[0];
Dataset<Row>[] splits = dataset.randomSplit(new double[]{0.7, 0.3});
Dataset<Row> training = splits[0];
Dataset<Row> test = splits[1];
调用ml中的模型:
RandomForestClassifier rf = new RandomForestClassifier();
RandomForestClassificationModel model = rf.fit(training);
设置参数:
rf.setMaxBins(5);
rf.setMaxDepth(10);
rf.setImpurity("gini");
rf.setNumTrees(5);
rf.setSeed(12345);
预测评估:
MulticlassClassificationEvaluator evaluator = new MulticlassClassificationEvaluator();
double accuracy = evaluator.evaluate(predictions);
double error = 1-accuracy;
返回结果:
List<Row> data = Arrays.asList(
RowFactory.create(accuracy),
RowFactory.create(error)
);
StructType schema = new StructType(new StructField[]{
createStructField("accuracy", DoubleType, false),
createStructField("error", DoubleType, false)
});
return this.sparkSession.createDataFrame(data,schema);