Python 操作 Kafka，生产者和消费者代码 Demo

import json

from kafka import KafkaConsumer

consumer = KafkaConsumer(‘test’, bootstrap_servers=[‘localhost:9092’],

value_deserializer=lambda m: json.loads(m.decode(‘ascii’)))

for message in consumer:

print (“%s:%d:%d: key=%s value=%s” % (message.topic, message.partition, message.offset, message.key, message.value))

key 同样支持 json 格式生产和消费，只需指定 key_serializer 和 key_deserializer。

消费者-读取最早可读消息

---

-- coding:utf-8 --

from kafka import KafkaConsumer

consumer = KafkaConsumer(‘test’, bootstrap_servers=[‘localhost:9092’], auto_offset_reset=‘earliest’)

for message in consumer:

print (“%s:%d:%d: key=%s value=%s” % (message.topic, message.partition, message.offset, message.key, message.value))

earliest 移到最早的可用消息，latest 最新的消息。

消费者-手动设置偏移量

---

-- coding:utf-8 --

from kafka import KafkaConsumer

from kafka.structs import TopicPartition

consumer = KafkaConsumer(‘test’, bootstrap_servers=[‘localhost:9092’])

获取test主题的分区信息

print consumer.partitions_for_topic(‘test’)