kafka c语言开发,librdkafka: librdkafka 是 Apache Kafka 的 C/C++ 开发包，提供生产者、消费者和管理客户端...

librdkafka - the Apache Kafka C/C++ client library

Copyright (c) 2012-2020, Magnus Edenhill.

librdkafka is a C library implementation of the

Apache Kafka protocol, providing Producer, Consumer

and Admin clients. It was designed with message delivery reliability

and high performance in mind, current figures exceed 1 million msgs/second for

the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

KAFKA is a registered trademark of The Apache Software Foundation and

has been licensed for use by librdkafka. librdkafka has no

affiliation with and is not endorsed by The Apache Software Foundation.

Features

Full Exactly-Once-Semantics (EOS) support

High-level producer, including Idempotent and Transactional producers

High-level balanced KafkaConsumer (requires broker >= 0.9)

Simple (legacy) consumer

Admin client

Compression: snappy, gzip, lz4, zstd

SSL support

SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM, OAUTHBEARER) support

Broker version support: >=0.8 (see Broker version compatibility)

Guaranteed API stability for C & C++ APIs (ABI safety guaranteed for C)

Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu

RPM package: librdkafka and librdkafka-devel

Gentoo package: dev-libs/librdkafka

Portable: runs on Linux, MacOS X, Windows, Solaris, FreeBSD, AIX, ...

Documentation

Introduction and manual in INTRODUCTION.md.

Configuration properties in

CONFIGURATION.md.

Statistics metrics in STATISTICS.md.

NOTE: The master branch is actively developed, use latest release for production use.

Installation

Installing prebuilt packages

On Mac OSX, install librdkafka with homebrew:

$brew installlibrdkafka

On Debian and Ubuntu, install librdkafka from the Confluent APT repositories,

see instructions here and then install librdkafka:

$apt installlibrdkafka-dev

On RedHat, CentOS, Fedora, install librdkafka from the Confluent YUM repositories,

instructions here and then install librdkafka:

$yum installlibrdkafka-devel

On Windows, reference librdkafka.redist NuGet package in your Visual Studio project.

For other platforms, follow the source building instructions below.

Installing librdkafka using vcpkg

You can download and install librdkafka using the vcpkg dependency manager:

# Install vcpkg if not already installed

$git clone https://github.com/Microsoft/vcpkg.git

$cdvcpkg

$./bootstrap-vcpkg.sh

$./vcpkg integrate install

# Install librdkafka

$vcpkg installlibrdkafka

The librdkafka package in vcpkg is kept up to date by Microsoft team members and community contributors.

If the version is out of date, please create an issue or pull request on the vcpkg repository.

Build from source

Requirements

The GNU toolchain

GNU make

pthreads

zlib-dev (optional, for gzip compression support)

libssl-dev (optional, for SSL and SASL SCRAM support)

libsasl2-dev (optional, for SASL GSSAPI support)

libzstd-dev (optional, for ZStd compression support)

NOTE: Static linking of ZStd (requires zstd >= 1.2.1) in the producer

enables encoding the original size in the compression frame header,

which will speed up the consumer.

Use STATIC_LIB_libzstd=/path/to/libzstd.a ./configure --enable-static

to enable static ZStd linking.

MacOSX example:

STATIC_LIB_libzstd=$(brew ls -v zstd | grep libzstd.a$) ./configure --enable-static

Building

./configure

# Or, to automatically install dependencies using the system's package manager:

# ./configure --install-deps

# Or, build dependencies from source:

# ./configure --install-deps --source-deps-only

make

sudo make install

NOTE: See README.win32 for instructions how to build

on Windows with Microsoft Visual Studio.

NOTE: See CMake instructions for experimental

CMake build (unsupported).

Usage in code

Refer to the examples directory for code using:

Producers: basic producers, idempotent producers, transactional producers.

Consumers: basic consumers, reading batches of messages.

Performance and latency testing tools.

Refer to the examples GitHub repo for code connecting to a cloud streaming data service based on Apache Kafka

Link your program with -lrdkafka (C) or -lrdkafka++ (C++).

Commercial support

Commercial support is available from Confluent Inc

Community support

Only the last official release is supported for community members.

File bug reports, feature requests and questions using

GitHub Issues

Questions and discussions are also welcome on the Confluent Community slack #clients channel.

Language bindings

See Powered by librdkafka for an incomplete list of librdkafka users.