Python Kafaka 生产,普通消费,群组消费示例

Python Kafaka 生产,普通消费,群组消费示例

# 安装模块
pip install kafka-python

1-生产数据

import json
import random
from time import sleep

from kafka import KafkaProducer

# json类型
# producer = KafkaProducer(bootstrap_servers=['192.168.0.99:9092'], value_serializer=lambda v: json.dumps(v).encode('utf-8'))
# string类型
producer = KafkaProducer(bootstrap_servers=['192.168.0.99:9092'], value_serializer=str.encode)

num = 1
while True:
    i = random.randint(2, 5)
    sleep(i)
    msg = "msg%d" % num
    producer.send('test', msg)
    num += 1

2-普通消费

from kafka import KafkaConsumer
# goup_id为消费组,同一组内
consumer = KafkaConsumer('test', bootstrap_servers=['192.168.0.99:9092'])
for msg in consumer:
    recv = "%s:%d:%d: key=%s value=%s" % (msg.topic, msg.partition, msg.offset, msg.key, msg.value)
    print(recv)

3-群组消费(指定group_id)

from kafka import KafkaConsumer
# goup_id为消费组,同一组内合计消费所有生产数据
consumer = KafkaConsumer('test', group_id='group1', bootstrap_servers=['192.168.0.99:9092'])
for msg in consumer:
    recv = "%s:%d:%d: key=%s value=%s" % (msg.topic, msg.partition, msg.offset, msg.key, msg.value)
    print(recv)

4-示例:多组消费

第一组只有一个人消费
![在这里插入图片描述](https://img-blog.csdnimg.cn/20201231092512361.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80Mjg3MzkyOA==,size_16,color_FFFFFF,t_70

第一组结果为: 一个人消费了所有数据

第二组有两个人消费


第二组结果为:两人合计消费了所有数据

5-总结

不同组可以同时消费同当前生产的数据
同一组内只有一个可以消费当前生产的数据