filter底层的实现

上篇:flatMap底层实现

第一种方式，重写filter方法

代码实现：

package cn._51doit.flink.day02;

import org.apache.flink.api.common.functions.FilterFunction;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

/**
 * Transformation的filter底层的实现
 * 实现效果：满足需求就输出，不满足就不做输出操作【有限的数据流】
 * 取偶数
 *
 */
public class FilterDemo3 {
    public static void main(String[] args) throws Exception {
        //查看本地的并行度
        StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(new Configuration());
        //定义元素
        DataStreamSource<Integer> nums = env.fromElements(1, 2, 3, 4, 5, 6, 7, 8, 9);
        SingleOutputStreamOperator<Integer> even = nums.filter(new FilterFunction<Integer>() {
            @Override
            public boolean filter(Integer value) throws Exception {
                return value % 2 == 0;
            }
        });
        even.print();
        env.execute();
    }

}

打印输出：

---

第二种方式，Lambda表达式的使用

package cn._51doit.flink.day02;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
/**
 * Transformation的filter底层的实现
 * 实现效果：满足需求就输出，不满足就不做输出操作【有限的数据流】
 * 取偶数
 */
public class FilterDemo02 {
    public static void main(String[] args) throws Exception {
        //查看本地的并行度
        StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(new Configuration());
        //定义元素
        DataStreamSource<Integer> nums = env.fromElements(1, 2, 3, 4, 5, 6, 7, 8, 9,10);

        SingleOutputStreamOperator<Integer> even = nums.filter(i -> i % 2 == 0);

        even.print();
        env.execute();
    }

}

打印输出：

---

第三种方式，filter底层的实现【重写processElement方法】

代码实现：

package cn._51doit.flink.day02;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;

/**
 * Transformation的filter底层的实现
 * 实现效果：满足需求就输出，不满足就不做输出操作【有限的数据流】
 * 取偶数
 */
public class FilterDemo03 {
    public static void main(String[] args) throws Exception {
        //查看本地的并行度
        StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(new Configuration());
        //定义元素
        DataStreamSource<Integer> nums = env.fromElements(1, 2, 3, 4, 5, 6, 7, 8, 9,10);

        SingleOutputStreamOperator<Integer> even = nums.transform("MyFilter", TypeInformation.of(Integer.class),
                new MyStreamFilter()
        );

        even.print();
        env.execute();
    }

    public static class MyStreamFilter extends AbstractStreamOperator<Integer> implements OneInputStreamOperator<Integer,Integer>{


        //重写processElement方法
        @Override
        public void processElement(StreamRecord<Integer> element) throws Exception {

            Integer value = element.getValue();
            if (value % 2 ==0){
                //符合条件输出
                output.collect(element);
            }
        }
    }
}

打印输出：

源码底层读解

（1）点击filter进去查看，有算子名字、返回类型，它需要返回一个clean，需要进行对闭包引用类型序列化的检测

（2）点击StreamFilter查看，发现它继承了AbstractUdfStreamOperator，必须需要传FilterFunction函数，并且实现OneInputStreamOperator

（3）真正的逻辑代码是这里，表示来一条数据处理一条，返回true或false

@Override
 public void processElement(StreamRecord<IN> element) throws Exception {
 if (userFunction.filter(element.getValue())) {
 output.collect(element);
   }
 }

返回true，用output.collect把element输出，返回false，不做输出操作