C++中实现Apache Kafka的消息发送和接收

在C++中实现Apache Kafka的消息发送和接收，通常需要使用Kafka的C/C++客户端库，如`librdkafka`，这是一个流行的、高性能的Kafka客户端，支持C和C++接口。下面我将提供一个简单的示例来说明如何使用`librdkafka`进行消息的发送和接收。

 

### 安装librdkafka

 

首先，确保你已经安装了`librdkafka`库。可以通过以下命令安装（以Unix系统为例）：

 

```sh

git clone https://github.com/edenhill/librdkafka.git

cd librdkafka

./configure

make

sudo make install

```

 

### 包含头文件和链接库

 

在你的C++代码中，包含必要的头文件，并链接`librdkafka`库：

 

```cpp

#include <librdkafka/rdkafkacpp.h>

 

// 编译时链接librdkafka

// g++ -o my_kafka_app my_kafka_app.cpp -lrdkafka++

```

 

### 发送消息示例

 

下面是一个简单的消息生产者示例，展示如何向Kafka主题发送消息：

 

```cpp

#include <iostream>

#include <string>

#include <librdkafka/rdkafkacpp.h>

 

class ExampleEventCb : public RdKafka::EventCb {

public:

    void event_cb(RdKafka::Event &event) override {

        std::cout << "Event: " << event.str() << std::endl;

    }

};

 

int main() {

    ExampleEventCb eventCb;

    RdKafka::Conf *conf = RdKafka::Conf::create(RdKafka::Conf::CONF_GLOBAL);

    std::string errstr;

 

    // 设置配置

    conf->set("bootstrap.servers", "localhost:9092", errstr);

    conf->set("queue.buffering.max.messages", "100000", errstr);

    conf->set("dr_cb", &eventCb, errstr); // 设置回调用于处理交付报告

 

    // 创建生产者实例

    RdKafka::Producer *producer = RdKafka::Producer::create(conf, errstr);

    if (!producer) {

        std::cerr << "Failed to create producer: " << errstr << std::endl;

        return 1;

    }

 

    // 发送消息

    std::string msg = "Hello, Kafka!";

    RdKafka::ErrorCode resp = producer->produce("my-topic", RdKafka::Topic::PARTITION_UA, RdKafka::Producer::RK_MSG_COPY, const_cast<char*>(msg.c_str()), msg.size(), NULL, 0, NULL);

    if (resp != RdKafka::ERR_NO_ERROR) {

        std::cerr << "Failed to produce message: " << RdKafka::err2str(resp) << std::endl;

    } else {

        std::cout << "Message sent: " << msg << std::endl;

    }

 

    // 清理

    delete conf;

    delete producer;

 

    return 0;

}

```

 

### 接收消息示例

 

下面是一个简单的消费者示例，展示如何从Kafka主题接收消息：

 

```cpp

#include <iostream>

#include <string>

#include <librdkafka/rdkafkacpp.h>

 

class ExampleConsumerCb : public RdKafka::ConsumeCb {

public:

    void consume_cb(RdKafka::Message &message, void *opaque) override {

        if (message.err() == RdKafka::ERR_NO_ERROR) {

            std::cout << "Consumed message [" << message.topic_name() << "][" << message.partition() << "] at offset " << message.offset()

                      << ": " << std::string(static_cast<const char*>(message.payload()), message.len()) << std::endl;

        } else {

            std::cerr << "Consumer error: " << message.errstr() << std::endl;

        }

    }

};

 

int main() {

    ExampleConsumerCb consumerCb;

    RdKafka::Conf *conf = RdKafka::Conf::create(RdKafka::Conf::CONF_GLOBAL);

    std::string errstr;

 

    // 设置配置

    conf->set("bootstrap.servers", "localhost:9092", errstr);

    conf->set("group.id", "my-consumer-group", errstr);

    conf->set("enable.auto.commit", "false", errstr); // 根据需要调整自动提交偏移量的设置

    conf->set("auto.offset.reset", "earliest", errstr); // 在没有偏移量或偏移量无效时从最早的偏移量开始消费

 

    // 创建消费者实例

    RdKafka::Consumer *consumer = RdKafka::Consumer::create(conf, errstr);

    if (!consumer) {

        std::cerr << "Failed to create consumer: " << errstr << std::endl;

        return 1;

    }

 

    // 订阅主题

    RdKafka::ErrorCode resp = consumer->subscribe("my-topic");

    if (resp != RdKafka::ERR_NO_ERROR) {

        std::cerr << "Failed to subscribe to topic: " << RdKafka::err2str(resp) << std::endl;

        return 1;

    }

 

    // 消费消息

    while (true) {

        RdKafka::Message *msg = consumer->consume(1000 /* timeout in ms */);

        consumerCb.consume_cb(*msg, NULL);

        delete msg;

    }

 

    // 清理

    consumer->unsubscribe();

    delete consumer;

    delete conf;

 

    return 0;

}

```

 

注意：上述示例代码仅作为基本演示用途，实际应用中应考虑更多的异常处理、资源管理和性能优化措施。