Storm整合Kafka

先丢个官网链接
本章博客依赖官网
第一步，先放依赖：
//为了防止冲突，我们用exclusion将log4j依赖排除

<dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-clients</artifactId>
      <version>0.9.0.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.11</artifactId>
      <version>0.9.0.0</version>
      <exclusions>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    <dependency>
      <groupId>org.apache.curator</groupId>
      <artifactId>curator-framework</artifactId>
      <version>2.12.0</version>
    </dependency>
    <dependency>
      <groupId>org.apache.curator</groupId>
      <artifactId>curator-recipes</artifactId>
      <version>2.12.0</version>
    </dependency>

同时也要将上面的storm-core里的依赖排除

<dependency>
      <groupId>org.apache.storm</groupId>
      <artifactId>storm-core</artifactId>
      <version>${storm.version}</version>
      <exclusions>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>log4j-over-slf4j</artifactId>
        </exclusion>
        <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-api</artifactId>
        </exclusion>
      </exclusions>
    </dependency>

接下来我们先写一个bolt方法

public class LogProcessBlot extends BaseRichBolt {
	//需要发送所以将collector定义出去
    OutputCollector collector;
    @Override
    //初始化
    public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
        this.collector = collector;
    }

    @Override
    public void execute(Tuple input) {
    	//因为kafka发送过来以byte方式，名为bytes，所以我们需要用getBinaryByField函数
        byte[] bytes = input.getBinaryByField("bytes");
        //返回是一个字节数组，我们把它转成string类型
        String s = new String(bytes);
        //这边打印测试
        System.out.println("value : "+s.toString());
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {

    }
}

再写main方法,spout方法可以写在main方法里

public static void main(String[] args) {
		//这边官网可供参考
		//设置zookeeper端口
        BrokerHosts hosts = new ZkHosts("localhost:2181");
        String topic = "project";
        String zkRoot = "/" + topic;
        String s = UUID.randomUUID().toString();
        //新建一个SpoutConfig，里面4个参数
        //1.zookeeper端口
        //2.topic名
        //3.zookeeperRoot地址
        //一个id,这边我们取一个随机数
        SpoutConfig spoutConfig = new SpoutConfig(hosts, topic,zkRoot,s);
		//这边创建一个KafkaSpout，直接将上面的SpoutConfig丢进去
        KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);

		//新建一个builder
        TopologyBuilder builder = new TopologyBuilder();
        //builder操作连接Spout和Bolt
        builder.setSpout("KafkaSpout",kafkaSpout);
        builder.setBolt("LogProcessBlot",new LogProcessBlot()).shuffleGrouping("KafkaSpout");
		
		//这边我们以本地方式进行测试
        LocalCluster cluster = new LocalCluster();
        cluster.submitTopology("StormKafkaTopology",new Config(),builder.createTopology());
    }

测试之前我们要先开启zookeeper

./bin/zkServer.sh start

然后打开Kafka

./bin/kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties 

创建topic

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic project

开启生产者端

./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic peojext

然后我们就可以开始测试了，开启main方法，然后往生产者端生产数据，看控制台

数据被storm捕获，这样我们的Storm整合Kafka就完成了
如果你想将他偏移量设置为当前开始，只需要将下列代码设置到spoutconfig中

spoutConfig.startOffsetTime = kafka.api.OffsetRequest.LatestTime();