【C++类库】：modern-cpp-kafka

KAI-萧山

已于 2022-07-31 12:14:54 修改

阅读量901

点赞数

分类专栏：【C/C++ 类库】笔记文章标签： kafka c++ 分布式

于 2022-07-31 12:12:31 首次发布

本文链接：https://blog.csdn.net/kai15058157346/article/details/124558497

版权

【C/C++ 类库】笔记专栏收录该内容

4 篇文章 1 订阅

订阅专栏

【github】：librdkafka
【github】：modern-cpp-kafka
【Confluent】：librdkafka文档
 【Confluent】：modern-cpp-kafka文档

1、librdkafka示例

依赖库：

The GNU toolchain
GNU make
pthreads
zlib-dev (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)
libzstd-dev (optional, for ZStd compression support)

安装依赖库：

// zlib、zlib.dev
sudo apt install zlib1g				//运行库
sudo apt-get install zlib1g.dev		//开发包
sudo apt-get install libssl-dev
sudo apt-get install libsasl2-dev
sudo apt-get install libzstd-dev

VS2019 创建 Linux工程，生成数据：

// https://github.com/edenhill/librdkafka/blob/master/examples/producer.cpp
#include "src-cpp/rdkafkacpp.h"

#include <cstdio>
#include <iostream>

int main(int argc, char** argv) 
{
    if (argc != 3) 
    {
        std::cerr << "Usage: " << argv[0] << " <brokers> <topic>\n";
        return 1;
    }

    std::string brokers = argv[1];		//192.168.146.128:9092
    std::string topic = argv[2];		//twice

    // Create configuration object
    RdKafka::Conf* conf = RdKafka::Conf::create(RdKafka::Conf::CONF_GLOBAL);

    std::string errstr;

    // Set bootstrap broker(s).
    conf->set("bootstrap.servers", brokers, errstr);

    // Set the delivery report callback.
    // The callback is only triggered from ::poll() and ::flush().
    struct ExampleDeliveryReportCb : public RdKafka::DeliveryReportCb 
    {
        void dr_cb(RdKafka::Message& message) 
        {
            /* If message.err() is non-zero the message delivery failed permanently for the message. */
            if (message.err())
                std::cerr << "% Message delivery failed: " << message.errstr() << std::endl;
            else
                std::cerr << "% Message delivered to topic " << message.topic_name() <<
                " [" << message.partition() << "] at offset " << message.offset() << std::endl;
        }
    } ex_dr_cb;

    conf->set("dr_cb", &ex_dr_cb, errstr);

    // Create a producer instance.
    RdKafka::Producer* producer = RdKafka::Producer::create(conf, errstr);

    delete conf;

    // Read messages from stdin and produce to the broker.
    std::cout << "% Type message value and hit enter to produce message. (empty line to quit)" 
    	<< std::endl;

    for (std::string line; std::getline(std::cin, line);) 
    {
        // Send/Produce message. This is an asynchronous call, 
        // on success it will only enqueue the message on the internal producer queue. 
    retry:
        RdKafka::ErrorCode err =
            producer->produce(
                /* Topic name */
                topic,
                /* Any Partition */
                RdKafka::Topic::PARTITION_UA,
                /* Make a copy of the value */
                RdKafka::Producer::RK_MSG_COPY /* Copy payload */,
                /* Value */
                const_cast<char*>(line.c_str()), line.size(),
                /* Key */
                NULL, 0,
                /* Timestamp (defaults to current time) */
                0,
                /* Message headers, if any */
                NULL,
                /* Per-message opaque value passed to delivery report */
                NULL);

        if (err != RdKafka::ERR_NO_ERROR) 
        {
            std::cerr << "% Failed to produce to topic " << topic << ": " <<
                RdKafka::err2str(err) << std::endl;

            if (err == RdKafka::ERR__QUEUE_FULL) 
            {
                // If the internal queue is full, wait for messages to be delivered and then retry. 
                producer->poll(1000/*block for max 1000ms*/);
                goto retry;
            }
        }
        else 
        {
            std::cout << "% Enqueued message (" << line.size() << " bytes) " <<
                "for topic " << topic << std::endl;
        } // A producer application should continually serve the delivery report queue 
        // by calling poll() at frequent intervals. producer->poll(0);

        if (line.empty()) break;
    }

    /* Wait for final messages to be delivered or fail. */
    /* wait for max 10 seconds */
    std::cout << "% Flushing final messages..." << std::endl; producer->flush(10 * 1000);

    if (producer->outq_len() > 0)
        std::cerr << "% " << producer->outq_len() << "message(s) were not delivered" << std::endl;

    delete producer;
}

缺陷：
1、手动资源管理 [原始指针] 而非 “资源获取即初始化” [RAII]
2、使用事件循环GOTO，而非推广的 Asio延续
3、子类所需回调RdKafka::DeliveryReportCb，一 lambda 就足够

优化：
1、智能指针管理浅拷贝消息的生命周期
2、封装隐藏内部队列管理和复杂的轮询规则
3、面向对象的接口替换函数的长参数列表

2、modern-cpp-kafka 生产者

在这里插入图片描述
AdminClient：支持管理和检查主题的 Kafka 管理客户端

KafkaProducer
ProducerRecordKafkaProducer：要发送的"消息类型"【由Topic、Partition、Key、Value和构成Headers】
Producer::Callback：用于提供请求完成的异步处理的回调方法；当发送到服务器的记录被确认时，将调用此方法
KafkaAsyncProducer：将记录异步发布到 Kafka 集群；每个send操作都需一 per-message Producer::Callback
KafkaSyncProducer：同步向Kafka集群发布记录；在交付完成前，send不返回
Producer::RecordMetadata：已被服务器确认的记录的元数据；
【含Topic, Partitions，Offset，KeySize，ValueSize，Timestamp和PersistedStatus】
AKafkaAsyncProducer将此元数据作为的输入参数传递Producer::Callback;
KafkaSyncProducer使用同步send方法返回元数据

同步生产者：

// https://github.com/morganstanley/modern-cpp-kafka/blob/main/examples/kafka_sync_producer.cc
#include "kafka/KafkaProducer.h"

#include <cstdio>
#include <iostream>
#include <string>

int main(int argc, char** argv)
{
    if (argc != 3) 
    {
        std::cerr << "Usage: " << argv[0] << " <brokers> <topic>\n";
        return 1;
    }

    std::string brokers = argv[1];
    kafka::Topic topic = argv[2];

    // Create configuration object
    kafka::Properties props({
      {"bootstrap.servers", brokers},
      {"enable.idempotence", "true"},	//确保消息按原始顺序仅成功发送一次
        });

    // Create a producer instance.
    kafka::KafkaSyncProducer producer(props);

    // Read messages from stdin and produce to the broker.
    std::cout << "% Type message value and hit enter to produce message. (empty line to quit)" 
    	<< std::endl;

    for (std::string line; std::getline(std::cin, line);) 
    {
        // The ProducerRecord doesn't own `line`, it is just a thin wrapper
        auto record = kafka::ProducerRecord(topic,
            kafka::NullKey,
            kafka::Value(line.c_str(), line.size()));

        // Send the message.
        try 
        {
            kafka::Producer::RecordMetadata metadata = producer.send(record);
            std::cout << "% Message delivered: " << metadata.toString() << std::endl;
        }
        catch (const kafka::KafkaException& e) 
        {
            std::cerr << "% Message delivery failed: " << e.error().message() << std::endl;
        }

        if (line.empty()) break;
    };

    // producer.close(); // No explicit close is needed, RAII will take care of it
}

区别：
1、RAII 用于生命周期管理
2、异常用于错误处理
3、轮询和队列管理现已隐藏

缺陷：
1、无法同时发多条消息，慢网络将骤降性能

异步生产者：

 // Create a producer instance.
kafka::KafkaAsyncProducer producer(props);

// Read messages from stdin and produce to the broker.
std::cout << "% Type message value and hit enter to produce message. (empty line to quit)" 
	<< std::endl;

// 优化:避免消息复制
for (auto line = std::make_shared<std::string>();
        std::getline(std::cin, *line);
        line = std::make_shared<std::string>())
{
    // The ProducerRecord doesn't own `line`, it is just a thin wrapper
    auto record = kafka::ProducerRecord(topic,
        kafka::NullKey,
        kafka::Value(line->c_str(), line->size()));

    // 发送消息, 若队列已满, 则将继续等待 [前提:发送/超时,消息会释放]
    producer.send(record,
        // The delivery report handler
        // Note: Here we capture the shared_pointer of `line`,
        //       which holds the content for `record.value()`.
        // It makes sure the memory block is valid until the lambda finishes.
        [line](const kafka::Producer::RecordMetadata& metadata, std::error_code ec) 
        {
            if (!ec)
                std::cout << "% Message delivered: " << metadata.toString() << std::endl;
            else
                std::cerr << "% Message delivery failed: " << ec.message() << std::endl; 
        });

    if (line->empty()) break;
};

最新版Kafka 生成 KafkaProducer，借助参数区分 KafkaSyncProducer 和 KafkaAsyncProducer

#include "kafka/KafkaProducer.h"

#include <cstdio>
#include <iostream>
#include <string>

using namespace kafka::clients;

int main(int argc, char** argv)
{
    if (argc != 3)
    {
        std::cerr << "Usage: " << argv[0] << " <brokers> <topic>\n";
        exit(1); // NOLINT
    }

    std::string brokers = argv[1];
    kafka::Topic topic = argv[2];

    try {
        // Create configuration object
        kafka::Properties props({
          {"bootstrap.servers", brokers},
          {"enable.idempotence", "true"},	//确保消息按原始顺序仅成功发送一次
            });

        // Create a producer instance.
        KafkaProducer producer(props);

        // Read messages from stdin and produce to the broker.
        std::cout << "% Type message value and hit enter to produce message. (empty line to quit)" 
        	<< std::endl;

        std::string line; 
        std::getline(std::cin, line);
        
        // The ProducerRecord doesn't own `line`, it is just a thin wrapper
        auto record = producer::ProducerRecord(topic,
            kafka::NullKey,
            kafka::Value(line.c_str(), line.size()));

        // Send the message.
        try
        {
            producer::RecordMetadata metadata = producer.syncSend(record);
            std::cout << "% Message delivered: " << metadata.toString() << std::endl;
        }
        catch (const kafka::KafkaException& e)
        {
            std::cerr << "% Message delivery failed: " << e.error().message() << std::endl;
        }
        // producer.close(); // No explicit close is needed, RAII will take care of it
    }
    catch (const kafka::KafkaException& e) 
    {
        std::cerr << "% Unexpected exception caught: " << e.what() << std::endl;
    }
}

在这里插入图片描述

3、modern-cpp-kafka 消费者

KafkaConsumer
ConsumerRecordKafkaConsumer：实例返回的消息类型。它包含Topic, Partition, Offset, Key, Value, Timestamp, 和Headers
KafkaAutoCommitConsumerpoll：在每个操作上自动提交先前轮询的偏移量
KafkaManualCommitConsumer：提供确认消息的手册commitAsync和commitSync方法

自动提交消费者：

//https://github.com/morganstanley/modern-cpp-kafka/blob/main/examples/kafka_auto_commit_consumer.cc
#include "kafka/KafkaConsumer.h"

#include <cstdio>
#include <iostream>
#include <string>

using namespace kafka::clients;

int main(int argc, char** argv)
{
    if (argc != 3) 
    {
        std::cerr << "Usage: " << argv[0] << " <brokers> <topic>\n";
        exit(argc == 1 ? 0 : 1); // NOLINT
    }

    std::string brokers = argv[1];
    kafka::Topic topic = argv[2];

	try 
	{
	    // Create configuration object
	    kafka::Properties props({
	      {"bootstrap.servers", brokers},
	      {"enable.auto.commit", "true"}
	    });
	
	    // Create a consumer instance.
	    KafkaConsumer consumer(props);
	
	    // Subscribe to topics
	    consumer.subscribe({ topic });
	
	    // Read messages from the topic.
	    std::cout << "% Reading messages from topic: " << topic << std::endl;
	    while (true) 
	    {
	        auto records = consumer.poll(std::chrono::milliseconds(100));
	        for (const auto& record : records) 
	        {
	            // In this example, quit on empty message
	            if (record.value().size() == 0) return 0;
	
	            if (!record.error()) 
	            {
	                std::cout << "% Got a new message..." << std::endl;
	                std::cout << "    Topic    : " << record.topic() << std::endl;
	                std::cout << "    Partition: " << record.partition() << std::endl;
	                std::cout << "    Offset   : " << record.offset() << std::endl;
	                std::cout << "    Timestamp: " << record.timestamp().toString() << std::endl;
	                std::cout << "    Headers  : " << kafka::toString(record.headers()) << std::endl;
	                std::cout << "    Key   [" << record.key().toString() << "]" << std::endl;
	                std::cout << "    Value [" << record.value().toString() << "]" << std::endl;
	            }
	            else 
	            {
	                // Errors are typically informational, thus no special handling is required
	                std::cerr << record.toString() << std::endl;
	            }
	        }
	    }
	    // consumer.close(); // No explicit close is needed, RAII will take care of it
	} catch (const kafka::KafkaException& e) 
	{
        std::cerr << "% Unexpected exception caught: " << e.what() << std::endl;
    }
}

在这里插入图片描述
优点：
1、自动提交调度，确保使用者崩溃，也不会确认未处理消息
在成功消费时，每次轮询之前【而非之后】提交其位置，有效确认上次轮询期间收到的消息

手动提交消费者：

//https://github.com/morganstanley/modern-cpp-kafka/blob/main/examples/kafka_manual_commit_consumer.cc
#include "kafka/KafkaConsumer.h"

#include <cstdio>
#include <iostream>
#include <string>

using namespace kafka::clients;

int main(int argc, char** argv)
{
    if (argc != 3) 
    {
        std::cerr << "Usage: " << argv[0] << " <brokers> <topic>\n";
        exit(argc == 1 ? 0 : 1); // NOLINT
    }

    std::string brokers = argv[1];
    kafka::Topic topic = argv[2];

    try 
    {

        // Create configuration object
        kafka::Properties props({
            {"bootstrap.servers", brokers},
        });

        // Create a consumer instance
        KafkaConsumer consumer(props);

        // Subscribe to topics
        consumer.subscribe({ topic });

        auto lastTimeCommitted = std::chrono::steady_clock::now();

        // Read messages from the topic
        std::cout << "% Reading messages from topic: " << topic << std::endl;
        bool allCommitted = true;
        bool running = true;
        while (running) 
        {
            auto records = consumer.poll(std::chrono::milliseconds(100));
            for (const auto& record : records) 
            {
                // In this example, quit on empty message
                if (record.value().size() == 0) 
                {
                    running = false;
                    break;
                }

                if (!record.error()) 
                {
                    std::cout << "% Got a new message..." << std::endl;
                    std::cout << "    Topic    : " << record.topic() << std::endl;
                    std::cout << "    Partition: " << record.partition() << std::endl;
                    std::cout << "    Offset   : " << record.offset() << std::endl;
                    std::cout << "    Timestamp: " << record.timestamp().toString() << std::endl;
                    std::cout << "    Headers  : " << kafka::toString(record.headers()) << std::endl;
                    std::cout << "    Key   [" << record.key().toString() << "]" << std::endl;
                    std::cout << "    Value [" << record.value().toString() << "]" << std::endl;

                    allCommitted = false;
                }
                else 
                {
                    std::cerr << record.toString() << std::endl;
                }
            }

            if (!allCommitted) 
            {
                auto now = std::chrono::steady_clock::now();
                if (now - lastTimeCommitted > std::chrono::seconds(1)) 
                {
                    // Commit offsets for messages polled
                    std::cout << "% syncCommit offsets: " << kafka::utility::getCurrentTime() << std::endl;
                    consumer.commitSync(); // or commitAsync()

                    lastTimeCommitted = now;
                    allCommitted = true;
                }
            }
        }
        // consumer.close(); // No explicit close is needed, RAII will take care of it
    }
    catch (const kafka::KafkaException& e) 
    {
        std::cerr << "% Unexpected exception caught: " << e.what() << std::endl;
    }
}