storm-kafka 整合框架

最新推荐文章于 2022-07-18 15:51:00 发布

菜鸟好好吃

最新推荐文章于 2022-07-18 15:51:00 发布

阅读量178

点赞数

分类专栏： JAVA 文章标签： kafka-storm

本文链接：https://blog.csdn.net/qq_35128300/article/details/95460762

版权

JAVA 专栏收录该内容

26 篇文章 0 订阅

订阅专栏

storm ,kafka 版本都为1.2.2

所需的配置文件

public class KafkaSpoutAllConfig {

    public static Properties kafkaStormConsumer()
    {
        Properties props = new Properties();
        props.put("bootstrap.servers", "SH04_TEST:5121,SH05_TEST:5121,SH06_TEST:5121");
        props.put("group.id", "saturn-mid-br");
        props.put("enable.auto.commit", "true");
        props.put("auto.commit.interval.ms", "1000");
        props.put("session.timeout.ms", "30000");
        props.put("auto.offset.reset", "latest");
        props.put("key.deserializer", "StringDeserializer.class.getName()");
        props.put("value.deserializer", "StringDeserializer.class.getName()");
        return props;
    }
    public static Properties kafkaStormProducer()
    {
        Properties props = new Properties();
        props.put("bootstrap.servers", "SH04_TEST:5121,SH05_TEST:5121,SH06_TEST:5121");
        props.put("acks", "all");
        props.put("retries", "0");
        props.put("batch.size", "4096");
        props.put("linger.ms", "1");
        props.put("buffer.memory", "40960");
        props.put("key.serializer", StringSerializer.class.getName());
        props.put("value.serializer", StringSerializer.class.getName());
        return props;
    }

    public static KafkaSpoutConfig<String,String> newKafkaSpoutConfig(List<String> topics)
    {
        return  KafkaSpoutConfig.builder("SH04_TEST:5121,SH05_TEST:5121,SH06_TEST:5121",topics)
                .setProp(ConsumerConfig.GROUP_ID_CONFIG, "saturn-mid-br")
                .setProp(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 200)
                .setRetry(getRetryService())  // 定义重试策略
                .setOffsetCommitPeriodMs(10000)  // 定时提交偏移量的时间间隔,默认是15s
                .setFirstPollOffsetStrategy(org.apache.storm.kafka.spout.KafkaSpoutConfig.FirstPollOffsetStrategy.LATEST)
                .setMaxUncommittedOffsets(250)
                .build();
    }

    private static KafkaSpoutRetryService getRetryService() {
        return new KafkaSpoutRetryExponentialBackoff(KafkaSpoutRetryExponentialBackoff.TimeInterval.microSeconds(500),
                KafkaSpoutRetryExponentialBackoff.TimeInterval.milliSeconds(2), Integer.MAX_VALUE, KafkaSpoutRetryExponentialBackoff.TimeInterval.seconds(10));
    }
}

Topology文件：

kafka-spout框架提供一个自身的spout ，建立Spout直接new KafkaSpout。

builder.setSpout("KafkaSpout",new KafkaSpout<String, String>(KafkaSpoutAllConfig.newKafkaSpoutConfig(topicList)) , 1);
        builder.setBolt("DispatcherBolt",new DisPatcherBolt(),2).shuffleGrouping("KafkaSpout");
        //sport MatchInfo
        builder.setBolt("SportProcessBolt",new SportProcessBolt(),2).shuffleGrouping("DispatcherBolt","SportItem");
        builder.setBolt("MatchInfoProcessBolt",new MatchInfoProcessBolt(),2).shuffleGrouping("SportProcessBolt","MatchInfoItem");