python脚本生产kafka消息问题处理两则

一开始脚本是这样的，本地调试无误于是传到测试执行机上运行

server = “xxx”
topic = “xxx.xxx.xxx”
testProducer = KafkaProducerUtil(server=server, topic=topic)
with testProducer.open():
msg = {
“xxx”: “xxx”
}
future = testProducer.produceMsg(msg)
future.get(timeout=10)
self.test_case.assert_that(future).is_not_none()

报告中出现报错

kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Timeout after waiting for 10 secs.
at File “/home/udev04/envs/venv/lib/python3.6/site-packages/kafka/producer/future.py”, line 62, in get [“Timeout after waiting for %s secs.” % (timeout,))]

比较明显是在监听消息发送时发生了超时，怀疑网络不通，联系了测试平台负责人开网解决。然后很快遇到了第二个问题

kafka.errors.NoBrokersAvailable: NoBrokersAvailable
at File “/home/udev04/envs/venv/lib/python3.6/site-packages/kafka/client_async.py”, line 846, in check_version [raise Errors.NoBrokersAvailable()]
at File “/home/udev04/envs/venv/lib/python3.6/site-packages/kafka/client_async.py”, line 221, in init [self.config[‘api_version’] = self.check_version(timeout=check_timeout)]
at File “/home/udev04/envs/venv/lib/python3.6/site-packages/kafka/producer/kafka.py”, line 347, in init [**self.config)]
at File “/home/udev04/envs/venv/lib/python3.6/site-packages/cmbatutils/KafkaUtil.py”, line 83, in _open [self.producer = _KafkaProducer(bootstrap_servers=self.server, acks=self.acks, **self.extra_configs)]

根据提示应该是与api_version有关，于是跟着这个配置参数读了kafka库中关于生产者的代码，发现它在初始化生产者时会检查api_version，如果为空则会通过kafkaClient取到最后一次使用的version值，如果取不到就会抛出上面的NoBrokersAvailable异常。没有继续往深了翻源码，猜测因为本地经常使用kafka工具，已经缓存了api_version可以取到所以不会出错，执行机因为刚开网之前也没人在上面跑过kafka相关脚本，没有这个缓存所以出错。为了避免麻烦，最后的解决方案就直接在调用生产者时传入api_version值

server = “xxx”
topic = “xxx.xxx.xxx”
testProducer = KafkaProducerUtil(server=server, topic=topic, api_version=(0,10,1))
with testProducer.open():
msg = {
“xxx”: “xxx”
}
future = testProducer.produceMsg(msg)
future.get(timeout=10)
self.test_case.assert_that(future).is_not_none()