kafka python 性能_Kafka python消费者以并行线程运行

我是python和kafka的新手.我有一个脚本,应该启动三个kafka消费者,等待来自这些消费者的消息,并做一些其他的事情.在这一点上,我甚至不知道我是否朝着正确的方向前进,所以任何帮助都将受到赞赏.

class MainClass():

def do_something_before(self):

# something is done here

def start_consumer(self):

consumer1_thread = threading.Thread(target=self.cons1, args=())

consumer2_thread = threading.Thread(target=self.cons2, args=())

consumer1_thread.daemon = True

consumer2_thread.daemon = True

consumer1_thread.start()

consumer2_thread.start()

def cons1(self):

consumer = KafkaConsumer(bootstrap_servers='localhost:9092',

auto_offset_reset='earliest')

consumer.subscribe(['my-topic'])

for message in consumer:

print(message.value)

def cons2(self):

consumer = KafkaConsumer(bootstrap_servers='localhost:9092',

auto_offset_reset='earliest')

consumer.subscribe(['my2-topic'])

for message in consumer:

print(message.value)

def keep_working(self):

# something is done here

if __name__ == 'main':

g = MainClass()

g.do_something_before()

g.keep_working()

最佳答案

我添加了2个消费者的python-kafka示例(基本上是两个python进程),你可以在这里找到它在github链接https://github.com/Shubhamgorde/kafka-python-app.

无法发布整个python文件,它有点大.

from multiprocessing import Process

def consumeData(topic):

try:

consumer = KafkaConsumer(topic, value_deserializer=lambda v:

binascii.unhexlify(v).decode('utf-8'))

except:

print("Error!!")

for msg in consumer:

msg=ast.literal_eval(msg.value)

if(msg[2] == 'C'):

performCreditOperation(msg)

elif (msg[2] == 'D'):

performDebitOperation(msg)

t1 = Process(target=consumeData, args=('Credit_transac',))

t2 = Process(target=consumeData, args=('Debit_transac',))

t1.start()

t2.start()