先丢个官网链接
本章博客依赖官网
第一步,先放依赖:
//为了防止冲突,我们用exclusion将log4j依赖排除
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>0.9.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.9.0.0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-framework</artifactId>
<version>2.12.0</version>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>2.12.0</version>
</dependency>
同时也要将上面的storm-core里的依赖排除
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>${storm.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>log4j-over-slf4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
</exclusions>
</dependency>
接下来我们先写一个bolt方法
public class LogProcessBlot extends BaseRichBolt {
//需要发送所以将collector定义出去
OutputCollector collector;
@Override
//初始化
public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
this.collector = collector;
}
@Override
public void execute(Tuple input) {
//因为kafka发送过来以byte方式,名为bytes,所以我们需要用getBinaryByField函数
byte[] bytes = input.getBinaryByField("bytes");
//返回是一个字节数组,我们把它转成string类型
String s = new String(bytes);
//这边打印测试
System.out.println("value : "+s.toString());
}
@Override
public void declareOutputFields(OutputFieldsDeclarer declarer) {
}
}
再写main方法,spout方法可以写在main方法里
public static void main(String[] args) {
//这边官网可供参考
//设置zookeeper端口
BrokerHosts hosts = new ZkHosts("localhost:2181");
String topic = "project";
String zkRoot = "/" + topic;
String s = UUID.randomUUID().toString();
//新建一个SpoutConfig,里面4个参数
//1.zookeeper端口
//2.topic名
//3.zookeeperRoot地址
//一个id,这边我们取一个随机数
SpoutConfig spoutConfig = new SpoutConfig(hosts, topic,zkRoot,s);
//这边创建一个KafkaSpout,直接将上面的SpoutConfig丢进去
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);
//新建一个builder
TopologyBuilder builder = new TopologyBuilder();
//builder操作连接Spout和Bolt
builder.setSpout("KafkaSpout",kafkaSpout);
builder.setBolt("LogProcessBlot",new LogProcessBlot()).shuffleGrouping("KafkaSpout");
//这边我们以本地方式进行测试
LocalCluster cluster = new LocalCluster();
cluster.submitTopology("StormKafkaTopology",new Config(),builder.createTopology());
}
测试之前我们要先开启zookeeper
./bin/zkServer.sh start
然后打开Kafka
./bin/kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties
创建topic
./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic project
开启生产者端
./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic peojext
然后我们就可以开始测试了,开启main方法,然后往生产者端生产数据,看控制台
数据被storm捕获,这样我们的Storm整合Kafka就完成了
如果你想将他偏移量设置为当前开始,只需要将下列代码设置到spoutconfig中
spoutConfig.startOffsetTime = kafka.api.OffsetRequest.LatestTime();