java spark dataset,如何将自定义Java类转换为Spark数据集

I can't figure out a way to convert a List of Test objects to a Dataset in Spark

This is my class:

public class Test {

public String a;

public String b;

public Test(String a, String b){

this.a = a;

this.b = b;

}

public List getList(){

List l = new ArrayList();

l.add(this.a);

l.add(this.b);

return l;

}

}

解决方案

Your code in the comments to create a DataFrame is correct. However, there is a problem with the way you define Test. You can create DataFrames using your code only from Java Beans. Your Test class is not a Java Bean. Once you fix that, you can use the following code to create a DataFrame:

Dataset dataFrame = spark.createDataFrame(listOfTestClasses, Test.class);

and these lines to create a typed Dataset:

Encoder encoder = Encoders.bean(Test.class);

Dataset dataset = spark.createDataset(listOfTestClasses, encoder);