python使用kafka

python使用kafka

1.安装包：

pip install kafka
或
pip install python-kafka

文档：
https://kafka-python.readthedocs.io/en/master/usage.html

2.生产者：

#coding:utf-8

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers=['192.168.137.51:9092']) #此处ip可以是多个['0.0.0.1:9092','0.0.0.2:9092','0.0.0.3:9092' ]

for i in range(3):
    msg = "msg%d" % i
    producer.send('mtest', msg)

producer.close()

3.消费者

from kafka import KafkaConsumer

consumer = KafkaConsumer('mtest', bootstrap_servers=['192.168.137.51:9092'])
for message in consumer:
    print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,
    message.offset, message.key,
    message.value))

定时拉取消息， poll

from kafka import KafkaConsumer