python kafkaproducer send_Python Producer可以通过外壳发送,但不能通过.py发送

I have a running and tested Kafka cluster, and am trying to use a Python script to send messages to the brokers. This works when I use the Python3 shell and call the producer method, however when I put these same commands into a python file and execute it - the script seems to hang.

I am using the kafka-python library for the consumer and producer. When I use the Python3 shell I can see the messages appear in the topic using Kafka GUI tool 2.0.4

I've tried various loops and statements in the python code, but nothing seems to make it 'run' to completion.

>>>from kafka import KafkaProducer

>>>producer = KafkaProducer(bootstrap_servers='BOOTSTRAP_SRV:9092')

>>>producer.send('MyTopic', b'Has this worked?')

>>>>>>

And this works and bytes appears in the broker topic data.

When I put the same code as above in a python .py file and execute with Python3 it completes, but no data is sent to Kafka broker.

No error shown either.

from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers='BOOTSTRAP_SRV:9092')

producer.send('MyTopic', b'Some Data to Check')

解决方案

As you can see, it returns a future.

Kafka clients will batch records, they don't immeadiately send one record at a time, and to make it do that, you will need to wait or flush the producer buffer so that it'll send before the app exits. In other words, the interactive terminal keeps the producer data in-memory, running in the background, and the other way discards that data

future = producer.send(...)

try:

record_metadata = future.get(timeout=10)

except KafkaError:

# Decide what to do if produce request failed...

log.exception()

pass

Or just put producer.flush(), if you don't care about the metadata or grabbing the future.

题目描述的是一个Python编程任务,涉及到数据库操作、数据转换以及消息队列技术的应用。以下是简要步骤: 1. **生产者程序 (mysql_producer.py)**: - 首先,你需要安装必要的库如`pymysql`(用于连接MySQL数据库)、`json`(处理JSON格式)和`kafka-python`(Kafka客户端)。 - 使用`pymysql`连接到student表所在的数据库,执行SQL查询获取数据。例如: ```python import pymysql from kafka import KafkaProducer def get_student_data(): conn = pymysql.connect(host='your_host', user='your_user', password='your_password', db='your_db') cursor = conn.cursor() query = "SELECT * FROM student" cursor.execute(query) rows = cursor.fetchall() conn.close() return rows producer = KafkaProducer(bootstrap_servers=['localhost:9092']) # 指定Kafka服务器地址 student_data = get_student_data() for row in student_data: json_row = json.dumps(row) # 将每行数据转化为JSON格式 producer.send('mysql_topic', key=None, value=json_row.encode('utf-8')) # 发送到指定topic ``` - 确保生产者关闭连接并发送完所有数据。 2. **消费者程序 (mysql_consumer.py)**: - 同样需要`kafka-python`库。这个程序会从'mysql_topic'接收数据,然后解析成字典或对象。 ```python from kafka import KafkaConsumer import json consumer = KafkaConsumer('mysql_topic', bootstrap_servers=['localhost:9092']) for message in consumer: raw_data = message.value.decode('utf-8') # 解码接收到的消息 data_dict = json.loads(raw_data) # 转换回Python字典 process_data(data_dict) # 处理接收到的数据 ``` 完成以上两步后,生产者会将数据库中的student表数据转换为JSON并发送Kafka Topic,而消费者可以从该Topic订阅并处理这些JSON数据。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值