组件 | 版本 | 下载地址 |
---|---|---|
JDK | 1.8 | https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html |
Hadoop | 2.7.7 | https://archive.apache.org/dist/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz |
KaFka | 2.0.0 | https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz |
Zookeeper | 3.4.5 | https://archive.apache.org/dist/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz |
JDK安装和Hadoop安装参考地址:
https://blog.csdn.net/su_mingyang/article/details/120838047?spm=1001.2014.3001.5501
1、安装Kafka集群
1.1、解压kafka
tar -xzvf /chinaskills/kafka_2.11-2.0.0.tgz -C /usr/local/src/
1.2、重命名为kafka
mv /usr/local/src/kafka_2.11-2.0.0 /usr/local/src/kafka
1.3、配置环境变量(当前用户生效)
vi /root/.bash_profile
配置内容:
export KAFKA_HOME=/usr/local/src/kafka
export PATH=$PATH:$KAFKA_HOME/bin
1.4、加载环境变量
source /root/.bash_profile
1.5、启动kafaka-zookeeper
zookeeper-server-start.sh -daemon /usr/local/src/kafka/config/zookeeper.properties
1.6、启动kafak服务
kafka-server-start.sh -daemon /usr/local/src/kafka/config/server.properties
1.7 查看jps进程
[root@master ~]# jps
38304 QuorumPeerMain
38662 Jps
38600 Kafka
2、测试kafka
2.1、创建topic
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
输出信息:
Created topic "test".
2.2 、生产数据
查看topic
kafka-topics.sh --list --zookeeper localhost:2181
输出信息:
test
运行生产者
kafka-console-producer.sh --broker-list localhost:9092 --topic test
写入数据:
>hello word
>hello word
>hive kafka flume
2.3 、新开一个终端进行消费数据
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
输出信息:
hello word
hello word
hive kafka flume
3、配置代理
3.1、复制server.properties文件
cp /usr/local/src/kafka/config/server.properties /usr/local/src/kafka/config/server-1.properties
cp /usr/local/src/kafka/config/server.properties /usr/local/src/kafka/config/server-2.properties
3.2、配置代理服务
**server-1.properties **
vi /usr/local/src/kafka/config/server-1.properties
配置内容
broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs-1
**server-2.properties **
vi /usr/local/src/kafka/config/server-2.properties
配置内容
broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/tmp/kafka-logs-2
3.3、 启动代理集群
kafka-server-start.sh -daemon /usr/local/src/kafka/config/server-1.properties
kafka-server-start.sh -daemon /usr/local/src/kafka/config/server-2.properties
3.4、查看jps进程
[root@master ~]# jps
38304 QuorumPeerMain
40992 Jps
40931 Kafka
39764 ConsoleProducer
38600 Kafka
40605 Kafka
3.5、创建一个3分区的主题
kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
输出信息:
Created topic "my-replicated-topic".
3.6、查看描述主题
kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
输出信息:
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2