java lambda spark_java – Apache Spark Lambda表达式 – 序列化问题

我试图在spark任务中使用lambda表达式,并抛出“

java.lang.IllegalArgumentException：无效的lambda反序列化”异常.当代码如“transform(pRDD-> pRDD.map(t-> t._2))”时抛出此异常.代码片段如下.

JavaPairDStream aggregate = pairRDD.reduceByKey((x,y)->x+y);

JavaDStream con = aggregate.transform(

(Function,JavaRDD>)pRDD-> pRDD.map(

(Function)t->t._2));

JavaPairDStream> & Serializable)pRDD-> pRDD.map(

(Function & Serializable)t->t._2));

以上两个选项没有奏效.好像我将对象“f”作为参数传递而不是lambda表达式“t-> t_.2”.有用.

Function f = new Function(){

@Override

public Integer call(Tuple2 paramT1) throws Exception {

return paramT1._2;

}

};

我可以知道将该函数表示为lambda表达式的正确格式是什么.

public static void main(String[] args) {

Function f = new Function(){

@Override

public Integer call(Tuple2 paramT1) throws Exception {

return paramT1._2;

}

};

JavaStreamingContext ssc = JavaStreamingFactory.getInstance();

JavaReceiverInputDStream lines = ssc.socketTextStream("localhost",9999);

JavaDStream words = lines.flatMap(s->{return Arrays.asList(s.split(" "));});

JavaPairDStream pairRDD = words.mapToPair(x->new Tuple2(x,1));

JavaPairDStreamx+y);

JavaDStream con = aggregate.transform(

(Function>)pRDD-> pRDD.map(

(Function)t->t._2));

//JavaDStream con = aggregate.transform(pRDD-> pRDD.map(f)); It works

con.print();

ssc.start();

ssc.awaitTermination();

}