Kafka 入门（四）-- Python Kafka Client 性能测试(2)

多线程模型1

测试代码：

1 import time
2 from threading import Thread
3 from datetime import datetime
4 from confluent_kafka import Consumer
5
6
7 class ChildThread(Thread):
8 def init(self, name, broker, topic):
9 Thread.init(self, name=name)
10 self.con = KafkaConsumerTool(broker, topic)
11
12 def run(self):
13 self.con.receive_msg()
14
15
16 class KafkaConsumerTool:
17 def init(self, broker, topic):
18 config = {
19 ‘bootstrap.servers’: broker,
20 ‘session.timeout.ms’: 30000,
21 ‘auto.offset.reset’: ‘earliest’,
22 ‘api.version.request’: False,
23 ‘broker.version.fallback’: ‘2.6.0’,
24 ‘group.id’: ‘test’
25 }
26 self.consumer = Consumer(config)
27 self.topic = topic
28
29 def receive_msg(self):
30 self.consumer.subscribe([self.topic])
31 print(datetime.now())
32 while True:
33 msg = self.consumer.poll(timeout=30.0)
34 print(msg)
35
36
37 if name == ‘main’:
38 thread_num = 10
39 threads = [ChildThread(“thread_” + str(i + 1), broker, topic) for i in range(thread_num)]
40
41 for i in range(thread_num):
42 threads[i].setDaemon(True)
43 for i in range(thread_num):
44 threads[i].start()
　　因为我使用的 Topic 共有8个分区，所以我分别测试了线程数在5个、8个和10个时消费50万数据所需要的时间，并计算每秒可消费的数据条数。

（2）多线程模型2

测试代码：

1 import time
2 from datetime import datetime
3 from confluent_kafka import Consumer
4 from threadpool import ThreadPool, makeRequests
5
6
7 class KafkaConsumerTool:
8 def init(self, broker, topic):
9 config = {
10 ‘bootstrap.servers’: broker,
11 ‘session.timeout.ms’: 30000,
12 ‘auto.offset.reset’: ‘earliest’,
13 ‘api.version.request’: False,
14 ‘broker.version.fallback’: ‘2.6.0’,
15 ‘group.id’: ‘mini-spider’
16 }
17 self.consumer = Consumer(config)
18 self.topic = topic
19
20 def receive_msg(self, x):
21 self.consumer.subscribe([self.topic])
22 print(datetime.now())
23 while True:
24 msg = self.consumer.poll(timeout=30.0)
25 print(msg)
26
27
28 if name == ‘main’:
29 thread_num = 10
30 consumer = KafkaConsumerTool(broker, topic)
31 pool = ThreadPool(thread_num)
32 for r in makeRequests(consumer.receive_msg, [i for i in range(thread_num)]):
33 pool.putRequest®
34 pool.wait()
USB Microphone https://www.soft-voice.com/
Wooden Speakers https://www.zeshuiplatform.com/
亚马逊测评 www.yisuping.cn
深圳网站建设www.sz886.com