kafka笔记

最新推荐文章于 2023-12-15 15:17:24 发布

xiexie1357

最新推荐文章于 2023-12-15 15:17:24 发布

阅读量715

点赞数

分类专栏：大数据

本文链接：https://blog.csdn.net/xiexie1357/article/details/113575680

版权

Kafka是一个分布式事件流处理平台，结合了发布订阅和消息队列的功能，提供高吞吐、持久性、容错性和可扩展性。其核心组件包括服务器（brokers）、生产者、消费者和消费者组。数据在Kafka中以事件的形式存储在主题中，每个主题被划分为多个分区，保证事件的顺序。Kafka通过副本和ack机制确保数据不丢失，适用于实时数据处理和大数据集成。

摘要由CSDN通过智能技术生成

简介

event streaming

central nervous system 中枢神经系统

event streaming is the practice of capturing data in real time from event sources like
databases,sensors,mobile devices,cloud services, and software applications in the form of streaming of events;
storing these event streams durably for later retrieval（检索）;manipulating, processing, and reacting to the event streams in real-time as well as retrospectively(可追溯);
and routing the event streams to different destination technologies as needed.

kafka is an event streaming platform

kafka combines three key capabilites so you can implement your use cases for event streaming end-to-end with a single battle-tested solution:

to public(write) and subscribe to(read) streaming of events, including continuous import/export of your data from other systems.
to store steams of events durably and reliably for as long as you want.
to process streams of events as they occur or retrospectively.

kafka是一个分布式发布-订阅消息系统。Kafka 适合离线和在线消息，消息保留在磁盘上，并在集群内复制以防止数据丢失。kafka构建在zookeeper同步服务之上。它与 Flink 和 Spark 有非常好的集成，应用于实时流式数据分析。
Kafka特点：

可靠性：具有副本及容错机制。
可扩展性：kafka无需停机即可扩展节点及节点上线。
持久性：数据存储到磁盘上，持久性保存。
性能：kafka具有高吞吐量。达到TB级的数据，也有非常稳定的性能。
速度快：顺序写入和零拷贝技术使得kafka延迟控制在毫秒级。

how does kafka work in a nutshell?

Kafka is a distributed system consisting of servers and clients that communicate via a high-performance TCP network protocal.

Servers:

kafka is run as a cluster of one or more servers that can span multiple datacenters or cloud regions.

Some of these servers form the storage layer, called the brokers.
Other servers run kafka connect to continuously import and export data as event streams to integrate kafka with your existing systems such as relational databases as well as other kafka cluster.

Clients:

They allow you to write distributed applications and microservices that read, write, and process streams of events in parallel, at scale, and in a fault-to