“Libraries for lz4 compression codec not found”的解决办法

升级Kafka服务器后，运行正常的Python程序突然报错：

    File "/usr/local/lib/python2.7/site-packages/kafka/record/legacy_records.py", line 240, in __iter__
    self._buffer = memoryview(self._decompress(key_offset))
  File "/usr/local/lib/python2.7/site-packages/kafka/record/legacy_records.py", line 181, in _decompress
    self._assert_has_codec(compression_type)
  File "/usr/local/lib/python2.7/site-packages/kafka/record/legacy_records.py", line 125, in _assert_has_codec
    "Libraries for {} compression codec not found".format(name))
kafka.errors.UnsupportedCodecError: UnsupportedCodecError: Libraries for lz4 compression codec not found

翻阅所有网上的方法，都提示升级kafka-python、pykafka可解决问题，结果折腾了半天，问题依旧。

后来翻阅到一篇“snappy codec”错误的帖子，解决办法是：

# on debian:
sudo apt-get install libsnappy-dev
#   followed by
pip install python-snappy

依样画葫芦，在CentOS执行相关命令：

#   查找相关的库包
yum search snappy
#   执行安装
yum install snappy-devel
#   查找python包
pip search snappy
#   安装python包，注意别选错了
pip install python-snappy

运行代码，依旧提示“Libraries for lz4 compression codec not found”错误，看来解码器安装得不对。

查找官方文档，找到如下一段话：

Compression

kafka-python supports gzip compression/decompression natively. To produce or consume lz4 compressed messages, you should install python-lz4 (pip install lz4). To enable snappy compression/decompression install python-snappy (also requires snappy library). See https://kafka-python.readthedocs.io/en/master/install.html#optional-snappy-install for more information.

再次安装之前的思路，执行依赖包的安装工作：

yum search lz4
#   找到lz4-devel.x86_64
yum install lz4-devel
# 查找python依赖
pip search lz4
#  执行安装
pip install lz4

再次运行，这次依旧没成功，但是错误提示变了，如下：

 File "/usr/local/lib/python2.7/site-packages/kafka/record/legacy_records.py", line 188, in _decompress
    uncompressed = lz4_decode_old_kafka(data.tobytes())
  File "/usr/local/lib/python2.7/site-packages/kafka/codec.py", line 282, in lz4_decode_old_kafka
    assert xxhash is not None

心都累了，这又是什么，查看codec.py源码，发现有如下代码：

try:
    import xxhash
except ImportError:
    xxhash = None

既然如此，那就是试试吧：

yum search xxhash
#   找到再运行
yum install xxhash-devel
pip search xxhash
#   找到再运行
pip install xxhash

最后一次运行代码(再不成功就要放弃python了)，终于成功！

结论

kafka-python消费或生产时，执行压缩需要安装独立的编码与解码包，如snappy、lz4。