python新消息_python kafka:在发布新消息之前,是否有方法阻止消费者关注kafka主题?...

我有一个消费者订阅了一个测试主题,其中生产者线程定期发布。我希望能够阻止消费者线程,直到打开新消息,然后处理该消息并再次开始等待。我最接近的是:

consumer = KafkaConsumer(topic_name, auto_offset_reset='latest',

bootstrap_servers=[localhost_],

api_version=(0, 10), consumer_timeout_ms=1000)

while True:

print(consumer.poll(timeout_ms=5000))

有没有更惯用的方法(或者这种方法有什么我看不到的严重问题)?

新来的卡夫卡对这种设计的一般建议非常欢迎。完整(运行)示例:

import time

from threading import Thread

import kafka

from kafka import KafkaProducer, KafkaConsumer

print('python-kafka:', kafka.__version__)

def publish_message(producer_instance, topic_name, key, value):

try:

key_bytes = bytes(str(key), encoding='utf-8')

value_bytes = bytes(str(value), encoding='utf-8')

producer_instance.send(topic_name, key=key_bytes, value=value_bytes)

producer_instance.flush()

except Exception as ex:

print('Exception in publishing message\n', ex)

localhost_ = 'localhost:9092'

def kafka_producer():

_producer = None

try:

_producer = KafkaProducer(bootstrap_servers=[localhost_],

api_version=(0, 10))

except Exception as ex:

print('Exception while connecting Kafka')

print(str(ex))

j = 0

while True:

publish_message(_producer, topic_name, value=j, key=j)

j += 1

time.sleep(5)

if __name__ == '__main__':

print('Running Producer..')

topic_name = 'test'

prod_thread = Thread(target=kafka_producer)

prod_thread.start()

consumer = KafkaConsumer(topic_name, auto_offset_reset='latest',

bootstrap_servers=[localhost_],

api_version=(0, 10), consumer_timeout_ms=1000)

# consumer.subscribe([topic_name])

while True:

print(consumer.poll(timeout_ms=5000))

python-kafka: 1.3.5