一、关于如何使用pykafka,请看这里
我想说的主要是pykafka消费消息时的问题
- 消费消息时我们很多时候希望不要重复消费,对已经消费过的信息进行舍弃
我查了很多解决方法都是这样的:
from pykafka import KafkaClient
client = KafkaClient(hosts="localhost:9092")
topic = client.topics['test']
consumer = topic.get_simple_consumer(
consumer_group='test1',
auto_commit_enable=True,
auto_commit_interval_ms=1,
consumer_id='test'
)
for x in consumer:
if x is not None:
print(x.value.decode('utf-8'))
- 但按照这样会报错:’SimpleConsumer’ object has no attribute ‘_consumer_group’
其实是因为kafka在传输的时候需要bytes,而不是str,所以在str上加上b标识就可以,如下
from pykafka import KafkaClient
client = KafkaClient(hosts="localhost:9092")
topic = client.topics[b'test']
consumer = topic.get_simple_consumer(
consumer_group=b'test1',
auto_commit_enable=True,
auto_commit_interval_ms=1,
consumer_id=b'test'
)
for x in consumer:
if x is not None:
print(x.value.decode('utf-8'))
这样就可以实现我们想要的不重复消费消息了