Golang 与Python 连接kafka

最新推荐文章于 2024-03-16 10:54:20 发布

我承包的鱼塘

最新推荐文章于 2024-03-16 10:54:20 发布

阅读量409

点赞数

分类专栏：大数据Spark 文章标签： kafka

本文链接：https://blog.csdn.net/weixin_39594447/article/details/86703900

版权

大数据Spark 专栏收录该内容

16 篇文章 0 订阅

订阅专栏

1. 安装

python	golang
pip install kafka-python	git clone https://github.com/edenhill/librdkafka.git cd librdkafka ./configure --prefix=/usr make && make install profile: export PKG_CONFIG_PATH=/usr/lib/pkgconfig go get -u github.com/confluentinc/confluent-kafka-go/kafka

python

golang

pip install kafka-python

git clone https://github.com/edenhill/librdkafka.git

cd librdkafka

./configure --prefix=/usr

make && make install

profile:

export PKG_CONFIG_PATH=/usr/lib/pkgconfig

go get -u github.com/confluentinc/confluent-kafka-go/kafka

编程Producer 连接

Python 编程 Golang编程

from kafka import KafkaProducer     import github.com/confluentinc/confluent-kafka-go/kafka
p = KafkaProducer(                   p,err := kafka.NewProducer(&kafka.ConfigMap{
 bootstrap_servers="master:9092"             "bootstrap.servers": "master"
)                                    })

Producer发送消息:

Python编程 Golang编程(异步发送)

msg = "send msg"                                msg := "send msg"
topic = "job"                                   topic := "job"
p.send(topic, msg)                              diverchan := make(chan kafka.Event)
                                                p.Produce(&kafka.Message{
                                                  TopicPartition: kafka.TopicPartition{
                                                          Topic: &topic,
                                                          Partition:kafka.PartitionAny
                                                  
                                                  },
                                                  Value: []byte(msg)
                                                 }, diverchan)
                                                 e := <- diverchan
                                                 et := e.(*kafka.Message)
                                                 if et.TopicPartition.Error == nil {
                                                            "send successfully"
                                                  }
                                                  close(diverchan)

Consumer编程:

Python编程 Golang编程

from kafka import KafkaConsumer                   c,er:=kafka.NewConsumer(&kafka.ConfigMap{
c = KafkaConsumer("job",                                    "bootstrap.servers": "master",
    bootstrap_servers=["master:9092"]                       "auto.offset.reset": "earliest"
)                                                  })  
for msg in c:                                      c.SubcribeTopics([]string{"job"},nil)
    print msg                                      while true:
                                                        data := c.ReadMessage(-1)
                                                        print data
                                                    c.Close()