kafka-python的基本使用及统计kafka消息总数

目录

1、kafka-python的安装及调用生产者，进行消息传入：

2、kafka-python调用消费者：

3、使用kafka-python，统计kafka的消息总数，这边其实是通过创建消费者对象，使用消费者内部获取offsets：

---

1、kafka-python的安装及调用生产者，进行消息传入：

# 安装包
pip install kafka-python

# 安装包安装后直接可以引用生产者的对象
>>> from kafka import KafkaProducer
>>> producer = KafkaProducer(bootstrap_servers='localhost:1234')
>>> for _ in range(100):
...     producer.send('foobar', b'some_message_bytes')


>>> # Block until all pending messages are sent
>>> producer.flush()

>>> # Block until a single message is sent (or timeout)
>>> producer.send('foobar', b'another_message').get(timeout=60)

>>> # Use a key for hashed-partitioning
>>> producer.send('foobar', key=b'foo', value=b'bar')

>>> # Serialize json messages
>>> import json
>>> producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8'))
>>> producer.send('fizzbuzz', {'foo': 'bar'})


>>> # Serialize string keys
>>> producer = KafkaProducer(key_serializer=str.encode)
>>> producer.send('flipflap', key='ping', value=b'1234')

>>> # Compress messages
>>> producer = KafkaProducer(compression_type='gzip')
>>> for i in range(1000):
...     producer.send('foobar', b'msg %d' % i)

2、kafka-python调用消费者：

# 直接引用kafka的消费者
>>> from kafka import KafkaConsumer
>>> consumer = KafkaConsumer('my_favorite_topic')
>>> for msg in consumer:
...     print (msg)

>>> # manually assign the partition list for the consumer
>>> from kafka import TopicPartition
>>> consumer = KafkaConsumer(bootstrap_servers='localhost:1234')
>>> consumer.assign([TopicPartition('foobar', 2)])
>>> msg = next(consumer)

>>> # Deserialize msgpack-encoded values
>>> consumer = KafkaConsumer(value_deserializer=msgpack.loads)
>>> consumer.subscribe(['msgpackfoo'])
>>> for msg in consumer:
...     assert isinstance(msg.value, dict)

3、使用kafka-python，统计kafka的消息总数，这边其实是通过创建消费者对象，使用消费者内部获取offsets：

# 使用kafka-python,同样可以获取统计到总数：
consumer = KafkaConsumer(bootstrap_servers='ip:port')
topic = 'test'
# 获取topic的分区
partition = list(consumer.partitions_for_topic(topic=topic))[0]
partitions = list(consumer.partitions_for_topic(topic=topic))

sum = 0
for partition in partitions:
    # print(partition)
    topic_partition = TopicPartition(topic=topic, partition=partition)
    consumer.assign([topic_partition])
    start_offset = consumer.beginning_offsets([topic_partition])[topic_partition]
    end_offset = consumer.end_offsets([topic_partition])[topic_partition]
    a = end_offset - start_offset
    sum = sum + a
print(sum)

结果：
44064

官方文档：

kafka-python · PyPI