flink 应用 keyby算子，增加数据迟延

胖胖胖胖胖虎

已于 2023-01-16 20:26:28 修改

阅读量217

点赞数

分类专栏： Flink 文章标签： flink java android

于 2023-01-13 17:55:23 首次发布

本文链接：https://blog.csdn.net/qq_15138049/article/details/128673297

版权

Flink 专栏收录该内容

27 篇文章 0 订阅

订阅专栏

Flink 添加 keyby 算子，增加数据

  		StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(4);
        //env.setBufferTimeout(0);
      	env.addSource(new SourceFunction<ABean>() {

            @Override
            public void run(SourceContext<ABean> ctx) throws Exception {
                while(true){
                	ctx.collect(ABean.builder().key("aaa").value(System.currentTimeMillis()).build());
                    Thread.sleep(100);
                }

            }

            @Override
            public void cancel() {

            }
        }).keyBy(new KeySelector<ABean, String>() {
            @Override
            public String getKey(ABean value) throws Exception {
                return value.getKey();
            }
        }).map(new RichMapFunction<ABean, ABean>() {
            @Override
            public ABean map(ABean value) throws Exception {

                log.warn("延迟时间：" + (System.currentTimeMillis() - value.getValue()));

                return value;
            }
        });
        env.execute("keyby 时间延迟");

在这里插入图片描述

The reason for this delay is that by adding that keyBy you are forcing a network shuffle along with serialization/deserialization. The reason the delay is so variable is because of the network buffering.

https://stackoverflow.com/questions/56819799/flink-keyby-adding-delay-how-can-i-reduce-this-latency

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
//
env.setBufferTimeout(0);

/**
     * Sets the maximum time frequency (milliseconds) for the flushing of the output buffers. By
     * default the output buffers flush frequently to provide low latency and to aid smooth
     * developer experience. Setting the parameter can result in three logical modes:
     *
     * <ul>
     *   <li>A positive integer triggers flushing periodically by that integer
     *   <li>0 triggers flushing after every record thus minimizing latency
     *   <li>-1 triggers flushing only when the output buffer is full thus maximizing throughput
     * </ul>
     *
     * @param timeoutMillis The maximum time between two output flushes.
     */
    public StreamExecutionEnvironment setBufferTimeout(long timeoutMillis) {
        if (timeoutMillis < -1) {
            throw new IllegalArgumentException("Timeout of buffer must be non-negative or -1");
        }

        this.bufferTimeout = timeoutMillis;
        return this;
    }