1、NoBrokersAvailable: NoBrokersAvailable-Kafka Error
解决方法:连接时指定版本
producer = KafkaProducer(bootstrap_servers=['localhost:9092'],api_version=(0,1,0))
2、AssertionError: Snappy decompression unsupported
解决方法:安装pip install python-snappy
可能需要先安装snappy-dev
# download snappy
http://pkgs.fedoraproject.org/repo/pkgs/snappy/
# install snappy
tar xvfz snappy-1.1.3.tar.gz
cd snappy-1.1.3
./configure --prefix=/usr/local/snappy
make
make install
3、python读取kafka指定分区、偏移量、负载均衡(从头读取,数据量大的话,可能没响应)
from kafka import KafkaConsumer
from kafka.structs import TopicPartition
consumer = KafkaConsumer(group_id='test', bootstrap_servers=['192.168.9.1:9002'])
consumer.assign([TopicPartition(topic='test', partition=0), TopicPartition(topic='test', partition=1)])
print(consumer.partitions_for_topic("test")) # 获取test主题的分区信息
print(consumer.assignment())
print(consumer.beginning_offsets(consumer.assignment()))
consumer.seek(TopicPartition(topic='test', partition=0), 0)
for msg in consumer:
recv = "%s:%d:%d: key=%s value=%s" % (msg.topic, msg.partition, msg.offset, msg.key, msg.value)
print(recv)
参考文献