我试图在spark任务中使用lambda表达式,并抛出“
java.lang.IllegalArgumentException:无效的lambda反序列化”异常.当代码如“transform(pRDD-> pRDD.map(t-> t._2))”时抛出此异常.代码片段如下.
JavaPairDStream aggregate = pairRDD.reduceByKey((x,y)->x+y);
JavaDStream con = aggregate.transform(
(Function,JavaRDD>)pRDD-> pRDD.map(
(Function)t->t._2));
JavaPairDStream> & Serializable)pRDD-> pRDD.map(
(Function & Serializable)t->t._2));
以上两个选项没有奏效.好像我将对象“f”作为参数传递而不是lambda表达式“t-> t_.2”.有用.
Function f = new Function(){
@Override
public Integer call(Tuple2 paramT1) throws Exception {
return paramT1._2;
}
};
我可以知道将该函数表示为lambda表达式的正确格式是什么.
public static void main(String[] args) {
Function f = new Function(){
@Override
public Integer call(Tuple2 paramT1) throws Exception {
return paramT1._2;
}
};
JavaStreamingContext ssc = JavaStreamingFactory.getInstance();
JavaReceiverInputDStream lines = ssc.socketTextStream("localhost",9999);
JavaDStream words = lines.flatMap(s->{return Arrays.asList(s.split(" "));});
JavaPairDStream pairRDD = words.mapToPair(x->new Tuple2(x,1));
JavaPairDStreamx+y);
JavaDStream con = aggregate.transform(
(Function>)pRDD-> pRDD.map(
(Function)t->t._2));
//JavaDStream con = aggregate.transform(pRDD-> pRDD.map(f)); It works
con.print();
ssc.start();
ssc.awaitTermination();
}