在Flink学习中,会常用到一些转换的算子,其中比如map,flatMap,filter,reduce等,进行数据转换,本篇是文章的第一部分,主要介绍简单的转换
- map算子,即对原来数据进行一个数据的转换,下面的式子仅仅是对value进行转换,完全也可以返回一个tuple
DataStream<Tuple2<String,Integer>> wordcount = env.fromElements(
new Tuple2<>("hello",1),
new Tuple2<>("hello",2),
new Tuple2<>("great",2));
wordcount.map(new MapFunction<Tuple2<String, Integer>, Integer>() {
@Override
public Integer map(Tuple2<String,Integer> value) throws Exception{
return value.f1 +1;
}
}).print();
env.execute("测试Map算子");
- reduce算子,对元素进行聚合,将输入的KeyedStream通过自定义的ReduceFunction进行聚合处理,自定义函数必须满足运算结合律和交换律,
DataStream<Tuple2<String,Integer>> wordcount = env.fromElements(
new Tuple2<>("hello",1),
new Tuple2<>("hello",2),
new Tuple2<>("great",2));
wordcount.keyBy(0).sum(1).print();
wordcount.keyBy(0).reduce(
new ReduceFunction<Tuple2<String, Integer>>() {
@Override
public Tuple2<String, Integer> reduce(Tuple2<String, Integer> t1, Tuple2<String, Integer> t2) throws Exception {
return Tuple2.of(t1.f0,t1.f1+t2.f1);
}
}
).print();
- filter算子,顾名思义即对DataStream进行过滤,满足一定条件之后的数据才进入新的DataStream中,代码如下,用了lambada表达式
DataStream<Tuple2<String,Integer>> wordcount = env.fromElements(
new Tuple2<>("hello",1),
new Tuple2<>("hello",2),
new Tuple2<>("great",2));
wordcount.filter((value) -> value.f1 % 2 ==0).print();
目前就写这些了,后续再丰富一下文档