[Hadoop] CentOS7 部署Kafka0.10.2.1（单节点）

1. 软件版本

• 操作系统：CentOS7.3
• Scala版本：2.11.12
• Zookeeper：zookeeper-3.4.5-cdh5.7.0
• Kafka版本：0.10.2.1（版本选择收spark straming+scala版本限制）

当前系统已经安装好scala和zookeeper。

 

2. Kafka安装

下载解压

[hadoop@hadoop01 software]# wget http://apache.mirrors.hoobly.com/kafka/0.10.2.1/kafka_2.11-0.10.2.1.tgz
[hadoop@hadoop01 software]$ tar -zxvf kafka_2.11-0.10.2.1.tgz -C ~/app/
[hadoop@hadoop01 software]$ cd ~/app/
[hadoop@hadoop01 app]$ ln -s kafka_2.11-0.10.2.1/ kafka

配置环境变量

[root@hadoop01 ~]# vi /etc/profile
export KAFKA_HOME=/home/hadoop/app/kafka
export PATH=$KAFKA_HOME/bin:$PATH

[root@hadoop01 ~]# source /etc/profile

修改kafka配置文件

[hadoop@hadoop01 kafka]$ vi config/server.properties 
broker.id=1
log.dirs=/data/kafka-logs
listeners=PLAINTEXT://192.168.1.8:9092
advertised.listeners=PLAINTEXT://192.168.1.8:9092
zookeeper.connect=192.168.1.8:2181

启动kafka

[hadoop@hadoop01 kafka]$ nohup kafka-server-start.sh config/server.properties &

[hadoop@hadoop01 kafka]$ netstat -nltp | grep 9092
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 192.168.1.8:9092       :::*                    LISTEN      5241/java           
[hadoop@hadoop01 kafka]$ 

# 停止使用命令
[hadoop@hadoop01 kafka]$ bin/kafka-server-stop.sh

 

3.Kafka基本操作

创建topic

[hadoop@hadoop01 kafka]$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
[hadoop@hadoop01 kafka]$

查看topic

[hadoop@hadoop01 kafka]$ bin/kafka-topics.sh --list --zookeeper localhost:2181
test
[hadoop@hadoop01 kafka]$ 

产生消息

[hadoop@hadoop01 kafka]$ bin/kafka-console-producer.sh --broker-list 192.168.1.8:9092 --topic test
hello kafka
hello zookeeper

消费消息

[hadoop@hadoop01 kafka]$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
hello kafka
hello zookeeper

查看topic描述信息

[hadoop@hadoop01 kafka]$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
Topic:test	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: test	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
[hadoop@hadoop01 kafka]$ 

删除topic

[hadoop@hadoop01 kafka]$ bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test

注意：kafka默认删除topic操作是disable掉的，要想enable此操作，要在kafka配置文件里面修改参数：

[hadoop@hadoop01 kafka]$ vi config/server.properties 
delete.topic.enable=true