python消费kafka_python生成kafka消息和消费kafka消息

1、创建连接kafka的对象

kafka_hosts = ["4.4.4.2","4.4.4.3","4.4.4.4"]

kafka_port = 6667

import time

kafka_producer = KafkaProducer(bootstrap_servers = ["{kafka_host1}:{kafka_port}".format(kafka_host1 = kafka_hosts[0],kafka_port = kafka_port),

"{kafka_host2}:{kafka_port}".format(kafka_host2 = kafka_hosts[1],kafka_port= kafka_port),

"{kafka_host3}:{kafka_port}".format(kafka_host3 = kafka_hosts[2],kafka_port = kafka_port)],max_request_size=101626282, buffer_memory=101626282,acks="all")

2、参数解释

max_request_size=101626282, buffer_memory=101626282

为了发送大文件

acks="all"

需要所有副本返回才可以

3、调用对象的send的方法发送数据

res = kafka_producer.send(topic="full-sight-insert", value=i)

print(res.get())

打印数据在kakfa中的位置

RecordMetadata(topic='full-sight-insert', partition=2, topic_partition=TopicPartition(topic='full-sight-insert', partition=2), offset=640782, timestamp=-1, checksum=-2084735199, serialized_key_size=-1, serialized_value_size=7122)

4、通过python消费kafka的消息

consumer = KafkaConsumer(topic,

auto_offset_reset='earliest',

bootstrap_servers=['10.1.4.12:9092'])

filename = os.path.join(filepath, str(topic) + ".txt")

while True:

message = consumer.poll(timeout_ms=120000)

if message:

for i in message.values():