python消费kafka逻辑处理导致cpu升高_Kafka-python 客户端导致的 cpu 使用过高，且无法消费消息的问题...

When no module exists to handle Snappy decompression, the KafkaConsumer returns no messages, rather than reporting the problem. This differs from the legacy Consumer API which provides a much more useful error message.

Background

I was attempting to fetch some data from a Kafka topic which was using snappy compression. No data was ever returned even though I knew data was being landed in the topic (confirmed with the Kafka CLI tools). This had me very confused.

>>> consumer = kafka.KafkaConsumer("test", bootstrap_servers=["svr:9092"])

>>> consumer.poll(5000)

{}

I then attempted to use the legacy consumer API which pointed me to the exact problem.

>>> client = kafka.SimpleClient("svr:9092")

>>> consumer.close()

>>> consumer = kafka.SimpleConsumer(client, "group", "test")

>>> for message in consumer:

... print(message)

...

Traceback (most recent call last):

File "", line 1, in

File "/usr/lib/python2.7/site-packages/kafka/consumer/simple.py", line 353, in __iter__

message = self.get_message(True, timeout)

File "/usr/lib/python2.7/site-packages/kafka/consumer/simple.py", line 305, in get_message

return self._get_message(block, timeout, get_partition_info)

File "/usr/lib/python2.7/site-packages/kafka/consumer/simple.py", line 320, in _get_message

self._fetch()

File "/usr/lib/python2.7/site-packages/kafka/consumer/simple.py", line 379, in _fetch

fail_on_error=False

File "/usr/lib/python2.7/site-packages/kafka/client.py", line 665, in send_fetch_request

KafkaProtocol.decode_fetch_response)

File "/usr/lib/python2.7/site-packages/kafka/client.py", line 295, in _send_broker_aware_request

for payload_response in decoder_fn(future.value):

File "/usr/lib/python2.7/site-packages/kafka/protocol/legacy.py", line 212, in decode_fetch_response

for partition, error, highwater_offset, messages in partitions

File "/usr/lib/python2.7/site-packages/kafka/protocol/legacy.py", line 219, in decode_message_set

inner_messages = message.decompress()

File "/usr/lib/python2.7/site-packages/kafka/protocol/message.py", line 121, in decompress

assert has_snappy(), 'Snappy decompression unsupported'

AssertionError: Snappy decompression unsupported

All I needed to do was install the python-snappy module to handle the decompression.

pip install python-snappy