从零开始搭建flink流式计算项目-3接入pulsar,使用pulsar-connector作为Source和Sink

电脑玩家柒柒

于 2024-03-31 12:41:31 发布

阅读量320

点赞数 7

文章标签： flink chrome 大数据

本文链接：https://blog.csdn.net/Kartist139/article/details/137196549

版权

接前一篇文章从零开始搭建flink流式计算项目-2小试牛刀-物联网场景下,如何实现设备采集参数监控报警功能

这里引入flink-connector-pulsarmaven坐标 , flink版本与pulsar-connector版本参考:

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-pulsar</artifactId>
            <version>4.1.0-1.18</version>
        </dependency>

这里新增一个生产者IotDataProducer,用于将消息生产到puslar中

public class IotDataProducer {
    static String serviceUrl = "pulsar://192.168.3.139:6650";
    static String topic = "persistent://flink/demo1/iot-source";

    public static void main(String[] args) throws Exception {

        PulsarClient pulsarClient = PulsarClient.builder()
                .serviceUrl(serviceUrl)
                .build();


        Producer<byte[]> producer = pulsarClient.newProducer(Schema.BYTES)
                .topic(topic)
                .create();

        while (true) {
            long ts = System.currentTimeMillis();
            IotData iotData = new IotData();
            iotData.setTimestamp(ts);
            iotData.setDeviceId(1);

            Map<String, Double> data = new HashMap<>();
            // 模拟数据的变化
            data.put("var1", RandomUtils.nextDouble());
            data.put("var2", RandomUtils.nextDouble());
            iotData.setData(data);
            producer.newMessage(Schema.BYTES)
                    .key("0")
                    .value(ObjectUtil.serialize(iotData))
                    .send();
            System.out.println(ts + " send iot data " + iotData);

            // 下一秒 0 毫秒的时间
            long timeMillis = System.currentTimeMillis();
            long nextMills = timeMillis - (timeMillis % 1000) + 1000;
            long sleepMills = nextMills - timeMillis;
            if (sleepMills > 0) {
                Thread.sleep(sleepMills);
            }
        }
    }
}

注意 ~ 这里的实体类 IotData 应实现Serializable接口

source

更新前一篇文章中 DataStreamSource<IotData> getIotStream(StreamExecutionEnvironment environment) 实现:

private static DataStreamSource<IotData> getIotStream(StreamExecutionEnvironment environment) {
        PulsarSource<IotData> source = PulsarSource.<IotData>builder()
                .setServiceUrl(pulsarServiceUrl)
                .setStartCursor(StartCursor.defaultStartCursor())
                .setTopics("persistent://flink/demo1/iot-source")
                .setDeserializationSchema(new AbstractDeserializationSchema<>() {
                    @Override
                    public IotData deserialize(byte[] message) throws IOException {
						IotData iotData = ObjectUtil.deserialize(message, IotData.class);
                        long timeMillis = System.currentTimeMillis();
                        System.out.println("delay:" + (timeMillis - iotData.getTimestamp()) + " = " + timeMillis + "-" + iotData.getTimestamp() + iotData);
                        return iotData;
                    }
                })
                .setSubscriptionName("iot-data-consumer")
                // 是否启用重试
                .setConfig(PulsarSourceOptions.PULSAR_RETRY_ENABLE, false)
                // 是否启用pulsar事务
                .setConfig(PulsarOptions.PULSAR_ENABLE_TRANSACTION, false)
                .build();

        return environment.fromSource(source, WatermarkStrategy.noWatermarks(), "alarm-message-srouce");
    }

sink

        alarmStream
        		.keyBy(d -> d.getDeviceId())
                .sinkTo(PulsarSink.<AlarmMessage>builder().setServiceUrl(pulsarServiceUrl)
                        .setTopics("persistent://flink/demo1/alarm-message")
                        .setSerializationSchema(new SerializationSchema<AlarmMessage>() {
                            @Override
                            public byte[] serialize(AlarmMessage element) {
                                return ObjectUtil.serialize(element);
                            }
                        }).setDeliveryGuarantee(DeliveryGuarantee.NONE)
                        // 如果是 exactly once ,一定要启用 pulsar transaction
                        .setConfig(PulsarOptions.PULSAR_ENABLE_TRANSACTION, false)
                        .build());

新建一个AlarmMessageConsumer用于接收sink的内容

public class AlarmMessageConsumer {

    static String serviceUrl = "pulsar://192.168.3.139:6650";
    static String topic = "persistent://flink/demo1/alarm-message";

    public static void main(String[] args) throws PulsarClientException {


        PulsarClient pulsarClient = PulsarClient.builder()
                .serviceUrl(serviceUrl)
                .build();


        System.out.println("start consumer");
        Consumer<byte[]> subscribe = pulsarClient.newConsumer(Schema.BYTES)
                .topic(topic)
                .subscriptionName("sub-name")
                .subscriptionInitialPosition(SubscriptionInitialPosition.Latest)
                .subscriptionType(SubscriptionType.Key_Shared)
                .subscribe();

        while (true) {
            try {
                Message<byte[]> message = subscribe.receive();
                MessageId messageId = message.getMessageId();
                AlarmMessage deserialize = ObjectUtil.deserialize(message.getValue(), AlarmMessage.class);
                long timeMillis = System.currentTimeMillis();
                System.out.println("delay:" + (timeMillis - deserialize.getTimestamp()) + " = " + timeMillis + "-" + deserialize.getTimestamp() + deserialize);

                subscribe.acknowledge(messageId);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }
}

启动IotDataProducer
输出

1711859596002 send iot data IotData(deviceId=1, timestamp=1711859596002, data={var2=1.4874470066475724E308, var1=8.768307315892181E307})
1711859597003 send iot data IotData(deviceId=1, timestamp=1711859597003, data={var2=8.524374320924697E307, var1=1.3400657409564972E308})
1711859598010 send iot data IotData(deviceId=1, timestamp=1711859598010, data={var2=1.2626688454103556E308, var1=1.2533802701073403E308})
1711859599002 send iot data IotData(deviceId=1, timestamp=1711859599002, data={var2=3.245087438200588E307, var1=1.2536063585892864E308})
1711859600012 send iot data IotData(deviceId=1, timestamp=1711859600012, data={var2=9.96467292821897E307, var1=7.113007341371873E307})
1711859601013 send iot data IotData(deviceId=1, timestamp=1711859601013, data={var2=1.3332675112179185E308, var1=1.7673592205934156E308})
1711859602003 send iot data IotData(deviceId=1, timestamp=1711859602003, data={var2=1.009336513276471E308, var1=9.623384612436023E307})
1711859603010 send iot data IotData(deviceId=1, timestamp=1711859603010, data={var2=7.237806299330377E307, var1=7.98211524294826E307})
1711859604011 send iot data IotData(deviceId=1, timestamp=1711859604011, data={var2=1.9813046690604265E307, var1=1.4557661543737139E308})
1711859605001 send iot data IotData(deviceId=1, timestamp=1711859605001, data={var2=2.1223495248921962E307, var1=3.948191317746565E306})
1711859606005 send iot data IotData(deviceId=1, timestamp=1711859606005, data={var2=9.342285346238023E306, var1=1.0044480538351647E308})
1711859607005 send iot data IotData(deviceId=1, timestamp=1711859607005, data={var2=1.203787261992374E308, var1=1.4427234811053151E308})
1711859608009 send iot data IotData(deviceId=1, timestamp=1711859608009, data={var2=1.0482444288783435E308, var1=1.1665217901453145E308})
1711859609013 send iot data IotData(deviceId=1, timestamp=1711859609013, data={var2=1.5516674739305947E308, var1=3.153307194590443E307})

启动flink-job
输出

delay:9612 = 1711859602616-1711859593004IotData(deviceId=1, timestamp=1711859593004, data={var2=9.815364221223E307, var1=5.310220544887899E307})
delay:8610 = 1711859602616-1711859594006IotData(deviceId=1, timestamp=1711859594006, data={var2=1.7484178752803745E308, var1=1.0461794592048479E308})
delay:7604 = 1711859602616-1711859595012IotData(deviceId=1, timestamp=1711859595012, data={var2=1.0138253557537254E308, var1=1.5523064239528138E308})
delay:6614 = 1711859602616-1711859596002IotData(deviceId=1, timestamp=1711859596002, data={var2=1.4874470066475724E308, var1=8.768307315892181E307})
delay:5613 = 1711859602616-1711859597003IotData(deviceId=1, timestamp=1711859597003, data={var2=8.524374320924697E307, var1=1.3400657409564972E308})
delay:4606 = 1711859602616-1711859598010IotData(deviceId=1, timestamp=1711859598010, data={var2=1.2626688454103556E308, var1=1.2533802701073403E308})
delay:3614 = 1711859602616-1711859599002IotData(deviceId=1, timestamp=1711859599002, data={var2=3.245087438200588E307, var1=1.2536063585892864E308})
delay:2604 = 1711859602616-1711859600012IotData(deviceId=1, timestamp=1711859600012, data={var2=9.96467292821897E307, var1=7.113007341371873E307})
delay:1603 = 1711859602616-1711859601013IotData(deviceId=1, timestamp=1711859601013, data={var2=1.3332675112179185E308, var1=1.7673592205934156E308})
delay:614 = 1711859602617-1711859602003IotData(deviceId=1, timestamp=1711859602003, data={var2=1.009336513276471E308, var1=9.623384612436023E307})
delay:9607 = 1711859612617-1711859603010IotData(deviceId=1, timestamp=1711859603010, data={var2=7.237806299330377E307, var1=7.98211524294826E307})
delay:8606 = 1711859612617-1711859604011IotData(deviceId=1, timestamp=1711859604011, data={var2=1.9813046690604265E307, var1=1.4557661543737139E308})
delay:7616 = 1711859612617-1711859605001IotData(deviceId=1, timestamp=1711859605001, data={var2=2.1223495248921962E307, var1=3.948191317746565E306})
delay:6612 = 1711859612617-1711859606005IotData(deviceId=1, timestamp=1711859606005, data={var2=9.342285346238023E306, var1=1.0044480538351647E308})
delay:5612 = 1711859612617-1711859607005IotData(deviceId=1, timestamp=1711859607005, data={var2=1.203787261992374E308, var1=1.4427234811053151E308})
delay:4609 = 1711859612618-1711859608009IotData(deviceId=1, timestamp=1711859608009, data={var2=1.0482444288783435E308, var1=1.1665217901453145E308})
delay:3605 = 1711859612618-1711859609013IotData(deviceId=1, timestamp=1711859609013, data={var2=1.5516674739305947E308, var1=3.153307194590443E307})

启动AlarmMessageConsumer输出日志

delay:9663 = 1711859602667-1711859593004AlarmMessage(deviceId=1, timestamp=1711859593004, alarmVar=var2, alarmValue=9.815364221223E307)
delay:8661 = 1711859602667-1711859594006AlarmMessage(deviceId=1, timestamp=1711859594006, alarmVar=var2, alarmValue=1.7484178752803745E308)
delay:7655 = 1711859602667-1711859595012AlarmMessage(deviceId=1, timestamp=1711859595012, alarmVar=var2, alarmValue=1.0138253557537254E308)
delay:6666 = 1711859602668-1711859596002AlarmMessage(deviceId=1, timestamp=1711859596002, alarmVar=var2, alarmValue=1.4874470066475724E308)
delay:5665 = 1711859602668-1711859597003AlarmMessage(deviceId=1, timestamp=1711859597003, alarmVar=var2, alarmValue=8.524374320924697E307)
delay:4658 = 1711859602668-1711859598010AlarmMessage(deviceId=1, timestamp=1711859598010, alarmVar=var2, alarmValue=1.2626688454103556E308)
delay:3666 = 1711859602668-1711859599002AlarmMessage(deviceId=1, timestamp=1711859599002, alarmVar=var2, alarmValue=3.245087438200588E307)
delay:2656 = 1711859602668-1711859600012AlarmMessage(deviceId=1, timestamp=1711859600012, alarmVar=var2, alarmValue=9.96467292821897E307)
delay:1655 = 1711859602668-1711859601013AlarmMessage(deviceId=1, timestamp=1711859601013, alarmVar=var2, alarmValue=1.3332675112179185E308)
delay:665 = 1711859602668-1711859602003AlarmMessage(deviceId=1, timestamp=1711859602003, alarmVar=var2, alarmValue=1.009336513276471E308)
delay:9677 = 1711859612687-1711859603010AlarmMessage(deviceId=1, timestamp=1711859603010, alarmVar=var2, alarmValue=7.237806299330377E307)
delay:8676 = 1711859612687-1711859604011AlarmMessage(deviceId=1, timestamp=1711859604011, alarmVar=var2, alarmValue=1.9813046690604265E307)
delay:7686 = 1711859612687-1711859605001AlarmMessage(deviceId=1, timestamp=1711859605001, alarmVar=var2, alarmValue=2.1223495248921962E307)
delay:6682 = 1711859612687-1711859606005AlarmMessage(deviceId=1, timestamp=1711859606005, alarmVar=var2, alarmValue=9.342285346238023E306)
delay:5682 = 1711859612687-1711859607005AlarmMessage(deviceId=1, timestamp=1711859607005, alarmVar=var2, alarmValue=1.203787261992374E308)
delay:4678 = 1711859612687-1711859608009AlarmMessage(deviceId=1, timestamp=1711859608009, alarmVar=var2, alarmValue=1.0482444288783435E308)
delay:3674 = 1711859612687-1711859609013AlarmMessage(deviceId=1, timestamp=1711859609013, alarmVar=var2, alarmValue=1.5516674739305947E308)

可以看到从 IotDataProducer 发出的消息, 在 flink-job中会存在一定的延迟, 而 flink-job的sink结果,在AlarmMessageConsumer处理相对比较实时

这可以通过flink-job中 checkPoint进行调节