回顾过去200多年,第一次工业革命蒸汽时代,第二次工业革命电、石油能源时代,第三次工业革命原子能、电子计算机时代,第四次工业革命我认为是互联网数据时代,谁掌握了数据,谁就掌握了财富与权力。在当今世界如何做到对大量数据收集分析成为世界话题,kafka插件应用而生,kafka做为流处理,被各个著名互联网公司广泛应用,列如今日头条的广告日志收集采用kafka插件
运行kafka,需要依赖 zookeeper,我们可以单独下载安装zookeeper集群,也可以利用kafka自带集群zookeeper集群
从kafka官网下载二进制编码包,下载地址:http://kafka.apache.org/downloads,我安装的是kafka_2.11-0.10.1.0.tgz版本
第一步:准备工作
1.准备三台机器192.168.138.130、192.168.138.131、192.168.138.132
2.安装jdk(我安装的是jdk-10_linux-x64_bin.tar.gz)
3.查看各个服务器防火墙是否关闭,如果没有关闭,要及时关闭防火墙
第二步:启动zookeeper需要修改zookeeper.properties配置文件
1.新增
tickTime=2000(心跳和超时)
initLimit=10(初始化连接时最长能忍受多少个心跳时间间隔数)
syncLimit=5(Leader与Follower之间发送消息,请求和应答时间长度,最长不能超过多少个tickTime 的时间长度)
server.1=192.168.138.130:2888:3888
server.2=192.168.138.131:2888:3888
server.3=192.168.138.132:2888:3888
2.修改
dataDir=/home/jiasd/datas/zookeeper(zookeeper数据存储路径)
3.执行命令启动zookeeper集群
bin/zookeeper-server-start.sh config/zookeeper.properties
出现错误日志
Unrecognized VM option ‘UseCompressedOops’
Error: Clould not create the Java Vritual Machine.
Error: A fatal exception has occurres . Program will exit.
解决方法:
找到bin/kafka-run-class.sh 文件,使用vim打开,我的这个版本是在115行
113 # JVM performance options
14 if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
115 KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRe mark -XX:+DisableExplicitGC -Djava.awt.headless=true"
116 fi
去掉-XX:+UseCompressedOops这个设置
第三步:启动kafka需要修改server.properties配置文件
1.修改
broker.id=0(每个broker保持唯一,可依次递增,另外两个机器我分别配置为broker.id=1,broker.id=1)
log.dirs=/home/jiasd/datas/kafka/logs(kafka数据存放地址)
num.partitions=3(每个topic的分区个数,大于等于broke数量)
zookeeper.connect=192.168.138.130:2181,192.168.138.131:2181,192.168.138.132:2181(连接zookeeper集群)
2.执行命令启动kafka集群
bin/kafka-server-start.sh config/server.properties
第四步:接下来我们验证一下kafka集群是否可以正常工作
1.创建一个topic
cd到192.168.138.130机器kafka的安装目录下,执行bin/kafka-topics.sh --create --zookeeper 192.168.138.130:2181 --replication-factor 3 --partitions 3 --topic test
2.查看topic
再执行bin/kafka-topics.sh --list --zookeeper 192.168.138.130:2181
3.创建一个生产者
再执行bin/kafka-console-producer.sh -broker-list 192.168.138.130:9092 -topic test
4.创建一个消费者
cd到192.168.138.131机器kafka的安装目录下,执行bin/kafka-console-consumer.sh -zookeeper 192.168.138.131:2181 -from-beginning -topic test
5.发送消息与接收消息演示
在第3步骤执行完后,输入数据i say hello world,我们192.168.138.131机器上收到i produce hello world
以上步骤完成了kafka集群搭建与测试,接下来我们通过java程序完成消息发送与接收,这里我用的是IDEA开发工具
1.建立一个maven项目
2.在pom.xml里配置kafka依赖版本包,如下:
<dependencies>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.0.0</version>
</dependency>
</dependencies>
3.编写producer与consumer程序
生产者
/**
* Created by lenovo on 2018/5/13.
*/
import java.util.Properties;
import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
public class ProducerDemo {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put("zk.connect", "192.168.138.131:2181,192.168.138.130:2181,192.168.138.132:2181");
props.put("metadata.broker.list","192.168.138.131:9092,192.168.138.130:9092,192.168.138.132:9092");
props.put("serializer.class", "kafka.serializer.StringEncoder");
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
for (int i = 1; i <= 100; i++) {
Thread.sleep(2000);
producer.send(new KeyedMessage<String, String>("test", "i say hello world"));
}
}
}
消费者
package kafka; /** * Created by lenovo on 2018/5/13. */ import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Properties; import kafka.consumer.Consumer; import kafka.consumer.ConsumerConfig; import kafka.consumer.ConsumerIterator; import kafka.consumer.KafkaStream; import kafka.javaapi.consumer.ConsumerConnector; import kafka.message.MessageAndMetadata; public class ConsumerDemo { private static final String topic = "test"; private static final Integer threads = 1; public static void main(String[] args) { Properties props = new Properties(); props.put("zookeeper.connect", "192.168.138.131:2181,192.168.138.130:2181,192.168.138.132:2181"); props.put("group.id", "1111"); props.put("auto.offset.reset", "smallest"); ConsumerConfig config = new ConsumerConfig(props); ConsumerConnector consumer =Consumer.createJavaConsumerConnector(config); Map<String, Integer> topicCountMap = new HashMap<String, Integer>(); topicCountMap.put(topic, 1); topicCountMap.put("testtopics", 1); Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap); List<KafkaStream<byte[], byte[]>> streams = consumerMap.get("test"); for(final KafkaStream<byte[], byte[]> kafkaStream : streams){ new Thread(new Runnable() { public void run() { for(MessageAndMetadata<byte[], byte[]> mm : kafkaStream){ String msg = new String(mm.message()); System.out.println(msg); } } }).start(); } } }