centos7 安装 kafka

环境 centos7 ，python 2.7 ， kafka 2.3.0，kafka-python 2.0.2

kafka-python 2.0.2 不支持 kafka 2.5.0 及以上版本，可以查看 pypi 官网信息
本文未做 kafka 高可用以及认证授权，只适合测试使用
kafka 原理可以参考文章 https://gitbook.cn/books/5ae1e77197c22f130e67ec4e/index.html

1、官网下载 kafka 安装包

2、启动 kafka 程序

[root@ansible002 ~]# tar -zxvf kafka_2.11-2.3.0.tgz
[root@ansible002 ~]# cd kafka_2.11-2.3.0

shell1 运行 zookeeper

[root@ansible002 kafka_2.11-2.3.0]# bin/zookeeper-server-start.sh config/zookeeper.properties

shell2 运行 kafka

[root@ansible002 kafka_2.11-2.3.0]# bin/kafka-server-start.sh config/server.properties

shell3 create topic 存储 events

[root@ansible002 kafka_2.11-2.3.0]# bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

查看 topic

[root@ansible002 kafka_2.11-2.3.0]# bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
Topic:quickstart-events	PartitionCount:1	ReplicationFactor:1	Configs:segment.bytes=1073741824
	Topic: quickstart-events	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

写 events 到 topic, ctrl + c 结束

[root@ansible002 kafka_2.11-2.3.0]# bin/kafka-console-producer.sh --topic  quickstart-events --broker-list localhost:9092
>Are you ok?
>I'm fine        
>Fuck you

读取 events, ctrl + c 结束

[root@ansible002 kafka_2.11-2.3.0]# bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
Are you ok?
I'm fine
Fuck you

停止 kafka，停止 bin/kafka-console-producer.sh 、bin/kafka-console-consumer.sh、bin/kafka-server-start.sh、bin/zookeeper-server-start.sh

如果需要清除 kafka 数据

rm -rf /tmp/kafka-logs /tmp/zookeeper

3、python 连接 kafka。使用 kafka 包不成功，使用 kafka-python 成功

[root@ansible002 kafka_2.12-2.5.0]# pip install kafka-python

生产者脚本

[root@ansible002 python-kafka]# cat send.py 
servers = ['127.0.0.1:9092']
from kafka import KafkaProducer
def producer_message():
    producer = KafkaProducer(bootstrap_servers=servers)
    for i in range(100):
        msg = "some_message_bytes " + str(i)
        producer.send('kafka-topic', bytes(msg.encode("utf-8")))
        print(msg)

if __name__ == '__main__':
    producer_message()

python send.py

消费者脚本

[root@ansible002 python-kafka]# cat recieve.py 
# -*- coding: utf-8 -*-
from kafka import KafkaConsumer
from kafka import TopicPartition

servers = ['127.0.0.1:9092']
# 消费方式1，默认消费所有分区的消息
def consumer_message1():
    consumer = KafkaConsumer('kafka-topic', 
                             bootstrap_servers=servers, 
                             group_id="kafka-group-id")
    # consumer = KafkaConsumer('kafka-topic', bootstrap_servers=servers)
    for msg in consumer:
        print(msg)

# 消费方式2, 指定消费分区
def consumer_message2():
    consumer = KafkaConsumer(bootstrap_servers=servers, 
                             group_id="kafka-group-id")
    consumer.assign([TopicPartition('kafka-topic', 0)])
    for msg in consumer:
        print(msg)

# 消费方式3，手动commit，生产中建议使用这种方式
def consumer_message3():
    consumer = KafkaConsumer(bootstrap_servers=servers,
                             consumer_timeout_ms=1000,
                             group_id="kafka-group-id",
                             enable_auto_commit=False)
    consumer.assign([TopicPartition('kafka-topic', 0)])
    for msg in consumer:
        print(msg)
        consumer.commit()

if __name__ == '__main__':
    print("消费方式1，默认消费所有分区的消息")
    consumer_message1()
    #print("消费方式2, 指定消费分区")
    #consumer_message2()
    #print("消费方式3，手动commit，生产中建议使用这种方式")
    #consumer_message3()

python recieve.py

http://kafka.apache.org/quickstart