CentOS7 kafka集群配置

节点1:10.116.186.6 jdk+zookeeper+kafka
节点2:10.116.186.7 jdk+zookeeper+kafka
节点3:10.116.186.8 jdk+zookeeper+kafka
OS：CentOS7.6 min x86_64

1. 软件安装

节点：节点1，节点2，节点3
说明：软件包（rpm）为我们企业内部自己打的包，主要是为了统一安装目录，日志目录，数据目录等。

yum list jdk zookeeper kafka
jdk.x86_64                   1.8.0_121-1.eju.el7            ejusofts
kafka.x86_64                 2.12-1.eju.el7                 ejusofts
zookeeper.x86_64             3.4.10-1.eju.el7               ejusofts

yum -y install jdk zookeeper kafka

2. zookeeper集群

2.1 配置

节点：节点1，节点2，节点3

grep -v -e ^$ -e ^# zoo.cfg 
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper/data
clientPort=2181
server.1=10.116.186.6:2888:3888
server.2=10.116.186.7:2888:3888
server.3=10.116.186.8:2888:3888

节点： 节点1 10.116.186.6

mkdir -p /var/lib/zookeeper/data
echo 1 > /var/lib/zookeeper/data/myid

节点： 节点2 10.116.186.7

mkdir -p /var/lib/zookeeper/data
echo 2 > /var/lib/zookeeper/data/myid

节点： 节点3 10.116.186.8

mkdir -p /var/lib/zookeeper/data
echo 3 > /var/lib/zookeeper/data/myid

# 2.2 启动节点，并验证集群

# 节点3,2,1 （顺序无所谓）根据实际安装目录执行命令
/opt/app/zookeeper/bin/zkServer.sh start

# 节点：1
/opt/app/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/bin/../conf/zoo.cfg
Mode: follower

# 节点：2
/opt/app/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/bin/../conf/zoo.cfg
Mode: follower

# 节点：3
/opt/app/zookeeper/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/app/zookeeper/bin/../conf/zoo.cfg
Mode: leader

2.3 连接测试

/opt/app/zookeeper/bin/zkCli.sh -server 10.116.186.6:2181
Connecting to 10.116.186.6:2181
2019-04-26 17:22:46,841 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2019-04-26 17:22:46,843 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=fangkafka-shylf-3
2019-04-26 17:22:46,843 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_121
2019-04-26 17:22:46,845 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-04-26 17:22:46,845 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/opt/app/install/jdk1.8.0_121/jre
2019-04-26 17:22:46,845 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/app/zookeeper/bin/../build/classes:/opt/app/zookeeper/bin/../build/lib/*.jar:/opt/app/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/app/zookeeper/bin/../lib/slf4j-api-1.6.1.jar:/opt/app/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/app/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/app/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/app/zookeeper/bin/../zookeeper-3.4.10.jar:/opt/app/zookeeper/bin/../src/java/lib/*.jar:/opt/app/zookeeper/bin/../conf:.:/opt/app/jdk/jre/lib/rt.jar:/opt/app/jdk/lib/dt.jar:/opt/app/jdk/lib/tools.jar
2019-04-26 17:22:46,845 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-04-26 17:22:46,845 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-04-26 17:22:46,845 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2019-04-26 17:22:46,845 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2019-04-26 17:22:46,846 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2019-04-26 17:22:46,846 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-957.10.1.el7.x86_64
2019-04-26 17:22:46,846 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2019-04-26 17:22:46,846 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2019-04-26 17:22:46,846 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/opt/app/install/zookeeper-3.4.10/conf
2019-04-26 17:22:46,847 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=10.116.186.6:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@799f7e29
Welcome to ZooKeeper!
2019-04-26 17:22:46,866 [myid:] - INFO  [main-SendThread(10.116.186.6:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server 10.116.186.6/10.116.186.6:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-04-26 17:22:46,954 [myid:] - INFO  [main-SendThread(10.116.186.6:2181):ClientCnxn$SendThread@876] - Socket connection established to 10.116.186.6/10.116.186.6:2181, initiating session
2019-04-26 17:22:46,983 [myid:] - INFO  [main-SendThread(10.116.186.6:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server 10.116.186.6/10.116.186.6:2181, sessionid = 0x16a58ed1b1c0000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: 10.116.186.6:2181(CONNECTED) 0] 

3. kafka集群

3.1 配置

grep -v -e ^$ -e ^# server.properties

broker.id=1            # broker.id=2  #broker.id=3     各节点不同值
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/var/lib/kafka/logs
num.partitions=3
num.recovery.threads.per.data.dir=1
log.retention.hours=168
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.116.186.6:2181,10.116.186.7:2181,10.116.186.8:2181
zookeeper.connection.timeout.ms=6000

mkdir -p /var/lib/kafka/logs

3.2 启动并验证

# 节点1,2，3
# 根据实际目录执行
/opt/app/kafka/bin/kafka-server-start.sh -daemon /opt/app/kafka/config/server.properties

jps
	22119 Kafka
	21739 QuorumPeerMain
	22239 Jps

3.3 topic验证

创建topic test
/opt/app/kafka/bin/kafka-topics.sh --zookeeper 10.116.186.6:2181 --create --replication-factor 2 --partitions 1 --topic test

[OUTPUT]
Created topic "test".


查看topic test
/opt/app/kafka/bin/kafka-topics.sh --zookeeper 10.116.186.6:2181 --list

[OUTPUT]
test

生产消息
/opt/app/kafka/bin/kafka-console-producer.sh --broker-list 10.116.186.6:9092 --topic test
[INPUT]
hello

消费消息[老的测试方式]
/opt/app/kafka/bin/kafka-console-consumer.sh --zookeeper 10.116.186.6:2181 --topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
[OUTPUT]
hello

消费消息[新方式]
/opt/app/kafka/bin/kafka-console-consumer.sh --bootstrap-server  10.116.186.6:9092 --topic test --from-beginning
hello