zxczb的博客

前置条件kafka 集群软件信息网盘下载地址链接：https://pan.baidu.com/s/1b5DjLQBmbz_8Nh7DyLj2BQ提取码：gin1install当前安装演示版本apache-flume-1.9.0-bin.tar.gzinstall# 解压tar -zxf apache-flume-1.9.0-bin.tar.gz -C /home/kafka/# 查看cd /home/kafka/apache-flume-1.9.0-bin/l

前置条件kafka 集群mysql软件信息源码地址https://github.com/smartloli/kafka-eagle下载地址http://download.kafka-eagle.org/网盘下载地址链接：https://pan.baidu.com/s/1b5DjLQBmbz_8Nh7DyLj2BQ提取码：gin1install当前安装演示版本kafka-eagle-bin-1.4.0.tar.gzinstall# 解压tar -zxf ka

kafkaSegmentsKafka的Topic被分为多个分区，分区是是按照Segments存储文件块(默认大小为1G, 超过1G则形成新的Segments)。分区日志是存储在磁盘上的日志序列，Kafka可以保证分区里的事件是有序的。其中Leader负责对应分区的读写、Follower负责同步分区的数据LEO：log end offset标识的是每个分区中最后一条消息的下一个位置，分区的每个副本都有自己的 LEOHW: high watermarker称为高水位线(同

producer before 上游package com.soul.kafka.level11;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.ProducerConfig;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.common.ser

consumer readcommitpackage com.soul.kafka.level10;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients

kafka的idempotence#mermaid-svg-l3ha4NcSSg88MEvH .label { font-family: 'trebuchet ms', verdana, arial; font-family: var(--mermaid-font-family); fill: #333; color: #333; }#mermaid-svg-l3ha4NcSSg88MEvH .label text { fill: #333; }#mermaid-svg

kafka的ack&retries#mermaid-svg-wCqAceYsBpZQzwcX .label { font-family: 'trebuchet ms', verdana, arial; font-family: var(--mermaid-font-family); fill: #333; color: #333; }#mermaid-svg-wCqAceYsBpZQzwcX .label text { fill: #333; }#mermaid

producerpackage com.soul.kafka.level07;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.ProducerConfig;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.common.serialization

consumerpackage com.soul.kafka.level06;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.K

producerpackage com.soul.kafka.level05;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.ProducerConfig;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.common.serialization

consumerpackage com.soul.kafka.level04;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.K

kafka topic DMLpackage com.soul.kafka.level03;import org.apache.kafka.clients.admin.*;import org.apache.kafka.common.KafkaFuture;import java.util.*;import java.util.concurrent.ExecutionException;public class _03TopicDML { public static void m

单机版安装安装jdk1.8+, 配置 JAVA_HOME配置主机名和IP映射(需特别注意!!!)关闭防火墙( CentOS7 )# 停止firewallsystemctl stop firewalld.service# 禁止firewall开机启动systemctl disable firewalld.service# 查看firewall状态( Active: inactive (dead) )systemctl status firewalld.service安装&启

kafkaWhat流处理平台scala和java编写高吞吐量的分布式发布订阅消息系统(Message Queue)系统间解耦, 异步通讯, 削峰填谷实时在线流处理(streamimg插件, 应用端), ps: 服务端流处理: strom, spark, flink)对于系统业务间的异步解耦用户用户注册消息队列发送短信业务30ms5ms异步流程总耗时为30ms短信业务不可用不影响用户注册系统间解耦30ms30ms流程总耗时为60ms用户用户注册消息队列发送短信业务对于日志埋点系统的削峰