142.5. Apache Kafka is a distributed publish-subscribe messaging system-CSDN博客

142.5.1. 安装 Kafka

142.5.1.1. 安装 Kafka用于开发与测试环境

如果你是开发或测试环境使用，可以使用内置 zookeeper

			
			
cd /usr/local/src
wget http://apache.communilink.net/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz
tar zxvf kafka_2.12-0.10.2.0.tgz
mv kafka_2.12-0.10.2.0 /srv/
cp /srv/kafka_2.12-0.10.2.0/config/server.properties{,.original}
echo "advertised.host.name=localhost" >> /srv/kafka_2.12-0.10.2.0/config/server.properties
ln -s /srv/kafka_2.12-0.10.2.0 /srv/kafka
/srv/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties
/srv/kafka/bin/kafka-server-start.sh /srv/kafka/config/server.properties

启动 Kafka 服务

			
/srv/kafka/bin/zookeeper-server-start.sh -daemon /srv/kafka/config/zookeeper.properties
/srv/kafka/bin/kafka-server-start.sh -daemon /srv/kafka/config/server.properties

-daemon 表示守护进程方式在后台启动

停止 Kafka 服务

/srv/kafka/bin/kafka-server-stop.sh
/srv/kafka/bin/zookeeper-server-stop.sh

142.5.1.2. 安装 Kafka 适用于 IDC

如果是生产环境安装脚本如下，独立安装zookeeper.

			
#!/bin/bash

cd /usr/local/src
wget http://apache.communilink.net/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz
tar zxvf zookeeper-3.4.9.tar.gz
cp zookeeper-3.4.9/conf/zoo_sample.cfg zookeeper-3.4.9/conf/zoo.cfg
vim zookeeper-3.4.9/conf/zoo.cfg
mv zookeeper-3.4.9 /srv/
ln -s /srv/zookeeper-3.4.9 /srv/zookeeper
#cd zookeeper-3.4.9
/srv/zookeeper/bin/zkServer.sh start

cd /usr/local/src
wget http://apache.communilink.net/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz
tar zxvf kafka_2.12-0.10.2.0.tgz
mv kafka_2.12-0.10.2.0 /srv/
cp /srv/kafka_2.12-0.10.2.0/config/server.properties{,.original}
echo "advertised.host.name=localhost" >> /srv/kafka_2.12-0.10.2.0/config/server.properties
ln -s /srv/kafka_2.12-0.10.2.0 /srv/kafka
/srv/kafka/bin/kafka-server-start.sh /srv/kafka/config/server.properties

启动 zookeeper

			
$ /srv/zookeeper/bin/zkServer.sh start

停止 zookeeper

			
$ /srv/zookeeper/bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /srv/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED

142.5.1.3. Kafka 日志

查看 server 日志

			
tailf /srv/kafka/logs/server.log

142.5.1.4. 检查 Kafka 线程

使用 jps 命令监控 Kafka 线程是否正确启动。

			
root@netkiller /srv/kafka/logs % jps | grep Kafka
32246 Kafka

142.5.2. 测试 Kafka

$ cd /srv/kafka

创建Topic

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test	
Created topic "test".

查看Topic

$ bin/kafka-topics.sh --list --zookeeper localhost:2181
test

启动Producer 生产消息

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message

启动Consumer 消费消息

$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
This is a message
This is another message

142.5.3. 配置 Kafka

142.5.3.1. 外网访问

默认 kafka对localhost提供访问，如果开放外面的IP进来你需要配置 config/server.properties

listeners = PLAINTEXT://147.189.135.55:9092

以及

advertised.host.name=147.189.135.55

142.5.3.2. group.id

查看 group.id 配置

# cat config/consumer.properties  | grep "group\.id"
group.id=test-consumer-group

142.5.4. 管理 Kafka

进入控制台

bin/zookeeper-shell.sh localhost:2181

删除Topic

$ /srv/kafka/bin/kafka-run-class.sh kafka.admin.TopicCommand --delete --topic kafkatopic --zookeeper localhost:2181

查看Topic 的 offset

$ /srv/kafka/bin/kafka-consumer-offset-checker.sh  --zookeeper localhost:2181 --topic kafkatopic --group consumer

142.5.5. FAQ

142.5.5.1. WARN Error while fetching metadata with correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

解决方法

			
echo "advertised.host.name=localhost" >> /srv/kafka/config/server.properties

142.5.5.2. Error while executing topic command : Replication factor: 1 larger than available brokers: 0.

			
root@VM_7_221_centos /srv/kafka % bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Error while executing topic command : Replication factor: 1 larger than available brokers: 0.
[2017-11-26 10:55:11,532] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0.
 (kafka.admin.TopicCommand$)

检查 broker.id 配置 broker.id 必须大于 0

			
root@netkiller /srv/kafka % cat config/server.properties | grep broker.id
broker.id=1

142.5.5.3. WARN Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

Kafka 在防火墙后面，防火墙上面配置 NAT 规则映射到服务器

			
# bind 任何IP地址
listeners=PLAINTEXT://:9092
# Wan IP 地址
advertised.host.name=223.207.161.225

	提示
	修改 advertised.host.name 后要删除 /tmp/kafka-logs 中的日志文件，否则无论如何你你都难以配置成功

			
rm -rf /tmp/kafka-logs