I can't figure out a way to convert a List of Test objects to a Dataset in Spark
This is my class:
public class Test {
public String a;
public String b;
public Test(String a, String b){
this.a = a;
this.b = b;
}
public List getList(){
List l = new ArrayList();
l.add(this.a);
l.add(this.b);
return l;
}
}
解决方案
Your code in the comments to create a DataFrame is correct. However, there is a problem with the way you define Test. You can create DataFrames using your code only from Java Beans. Your Test class is not a Java Bean. Once you fix that, you can use the following code to create a DataFrame:
Dataset dataFrame = spark.createDataFrame(listOfTestClasses, Test.class);
and these lines to create a typed Dataset:
Encoder encoder = Encoders.bean(Test.class);
Dataset dataset = spark.createDataset(listOfTestClasses, encoder);