大数据-消息队列：Kafka【分布式的基于push/pull模式的消息队列(Message Queue)；在流式计算中，Kafka一般用来缓存数据，Spark通过消费Kafka的数据进行计算】

一、Kafka集群的安装

1、在hadoop102服务器上安装Kafka

解压安装包，其中的2.11表示Kafka的Scala语言版本，0.11.0表示Kafka版本

[wyr@hadoop102 soft]$ tar -zxvf kafka_2.11-0.11.0.0.tgz  -C ../module/

修改解压后的文件名称

drwxrwxr-x.  9 wyr wyr 4096 Jan 31 14:45 flume
drwxr-xr-x. 11 wyr wyr 4096 Jan 31 10:43 hadoop-2.7.2
drwxrwxr-x.  9 wyr wyr 4096 Jan 30 19:27 hive
drwxr-xr-x.  8 wyr wyr 4096 Dec 13  2016 jdk1.8.0_121
drwxr-xr-x.  6 wyr wyr 4096 Jun 23  2017 kafka_2.11-0.11.0.0
drwxr-xr-x. 11 wyr wyr 4096 Jan 29 22:01 zookeeper-3.4.10
[wyr@hadoop102 module]$ mv kafka_2.11-0.11.0.0/ kafka
[wyr@hadoop102 module]$ ll
total 24
drwxrwxr-x.  9 wyr wyr 4096 Jan 31 14:45 flume
drwxr-xr-x. 11 wyr wyr 4096 Jan 31 10:43 hadoop-2.7.2
drwxrwxr-x.  9 wyr wyr 4096 Jan 30 19:27 hive
drwxr-xr-x.  8 wyr wyr 4096 Dec 13  2016 jdk1.8.0_121
drwxr-xr-x.  6 wyr wyr 4096 Jun 23  2017 kafka
drwxr-xr-x. 11 wyr wyr 4096 Jan 29 22:01 zookeeper-3.4.10
[wyr@hadoop102 module]$ 

2、配置

2.1 环境变量

1. Kafka使用Scala语言编写的，所以要保证安装Kafka的Linux服务器安装JDK，环境变量中有JAVA_HOME

2. Kafka会向Zookeeper中存储一些元数据，因此需要安装Zookeeper

2.2 修改配置文件

在/opt/module/kafka目录下创建datas文件夹(数据文件的存放目录)

[wyr@hadoop102 kafka]$ mkdir datas
[wyr@hadoop102 kafka]$ ll
total 56
drwxr-xr-x. 3 wyr wyr  4096 Jun 23  2017 bin
drwxr-xr-x. 2 wyr wyr  4096 Jun 23  2017 config
drwxrwxr-x. 2 wyr wyr  4096 Feb  1 16:12 datas
drwxr-xr-x. 2 wyr wyr  4096 Feb  1 15:57 libs
-rw-r--r--. 1 wyr wyr 28824 Jun 23  2017 LICENSE
-rw-r--r--. 1 wyr wyr   336 Jun 23  2017 NOTICE
drwxr-xr-x. 2 wyr wyr  4096 Jun 23  2017 site-docs
[wyr@hadoop102 kafka]$ 

编辑Kafka安装目录下的 /opt/module/kafka/config/server.properties 文件

#21行，cluster中的每台broker都需要有唯一的id号，必须为整数
broker.id=102
#24行，打开注释,此行代表允许手动删除kafka中的主题
delete.topic.enable=true
#63行，配置kafka存储的数据文件的存放目录
log.dirs=/opt/module/kafka/datas
#126行，配置连接的zk集群的地址
zookeeper.connect=hadoop101:2181,hadoop102:2181,hadoop103:2181

2.3 将hadoop102上的Kafka目录分发到hadoop101、hadoop103

[wyr@hadoop102 module]$ xsync.sh kafka

分别在hadoop101和hadoop103上修改配置文件/opt/module/kafka/config/server.properties中的broker.id=101、broker.id=103（注：broker.id不得重复）

3、启动/停止Kafka集群

Kafka依赖于Zookeeper，所以启动Kafka之前先启动Zookeeper，Kafka可以借助Zookeeper来动态地检测各个broker的上线与下线

3.1 在hadoop102中使用群起脚本群起Zookeeper集群

[wyr@hadoop102 ~]$ xcall.sh /opt/module/zookeeper-3.4.10/bin/zkServer.sh start
要执行的命令是/opt/module/zookeeper-3.4.10/bin/zkServer.sh start
----------------------------hadoop101----------------------------------
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
----------------------------hadoop102----------------------------------
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
----------------------------hadoop103----------------------------------
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[wyr@hadoop102 ~]$ xcall.sh /opt/module/zookeeper-3.4.10/bin/zkServer.sh status
要执行的命令是/opt/module/zookeeper-3.4.10/bin/zkServer.sh status
----------------------------hadoop101----------------------------------
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: leader
----------------------------hadoop102----------------------------------
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: follower
----------------------------hadoop103----------------------------------
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfg
Mode: follower
[wyr@hadoop102 ~]$ xcall.sh jps
要执行的命令是jps
----------------------------hadoop101----------------------------------
4181 QuorumPeerMain
4317 Jps
----------------------------hadoop102----------------------------------
4657 Jps
4515 QuorumPeerMain
----------------------------hadoop103----------------------------------
3804 QuorumPeerMain
3934 Jps
[wyr@hadoop102 ~]$ 

在hadoop101上登陆Zookeeper的客户端

[wyr@hadoop101 ~]$ /opt/module/zookeeper-3.4.10/bin/zkCli.sh -server hadoop101:2181
Connecting to hadoop101:2181
2021-02-01 16:29:24,726 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2021-02-01 16:29:24,728 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=hadoop101
2021-02-01 16:29:24,728 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_121
2021-02-01 16:29:24,729 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2021-02-01 16:29:24,729 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/opt/module/jdk1.8.0_121/jre
2021-02-01 16:29:24,729 [<