一、概述
消息队列
消息的概念
消息的是系统间通信的载体，是分布式应用不可获取的一部分。目前系统间发送消息有两种种类。
同步消息
即使消息：打电话、表达提交、WebService、Dubbo|SpringCloud
要求消息发送方和接受放必须同时在线，一般都需要和接收方建立会话。
异步消息
发送方不理会对方是否在线，一般不需要和接收方建立会话，在接受方上线后，一般会获取发送方发送的消息。
显而易见，可以看出消息队列就是使用的异步消息的模型。
消息队列
FIFO 先进先出
使用场景
异步消息
系统间解耦
削峰填谷
二、Kafka
Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.
kafka组件
Broker
每个Kafka Server称之为一个Broker，多个Broker可以组成Kafka Clutser
一个节点上可以有一个或者多个Broker，多个Broker连接到相同的ZK组成了Kafka集群。
Topic
Kafka是一个发布订阅消息系统
Topic就是消息类名，一个Topic中通常存放一类消息，每一个Topic都有一个或者多个订阅者，也就是消费者。
Producer 将消息推送到Topic，有订阅该Topic的Consumer从Topic中拉取消息。
Topic 与 Broker
一个Broker 上可以创建一个或者多个Topic，一个Topic可以同一集群下的多个Broker
Partition
kafka会为每个Topic维护多个分区（partititon），每个分区会映射到一个逻辑的日志（log）文件
一个Topic可以被划分为多个分区（自定义）
Partition Distrbution
对于同一个partiton，它所在任何一个broker，都有能扮演两种角色：leader、follower
每个partiton的Leader用于处理到该partiton的读写请求。
每个partiton的followers是用于异步的从它的leader中复制数据。
Kafka会动态维护一个与Leader保持一致的同步副本（in-sync replicas （ISR））集合，并且会将最新的同步副本（ISR ）集合持久化到zookeeper。如果leader出现问题了，就会从该partition的followers中选举一个作为新的leader。
Kafka 角色
Producer
Producer作为消息的生产者，在生产消息后需要将消息投送到指定的目的地（某个topic的某个partition）。Producer可以根据指定选择partiton的算法或者是随机方式来选择发布消息到哪个partiton
Consumer
在Kafka中，同样有consumer group的概念，它是逻辑上将一些consumer分组。因为每个kafka consumer是一个进程。所以一个consumer group中的consumers将可能是由分布在不同机器上的不同的进程组成的。Topic中的每一条消息可以被多个consumer group消费，然而每个consumer group内只能有一个consumer来消费该消息。所以，如果想要一条消息被多个consumer消费，那么这些consumer就必须是在不同的consumer group中。所以也可以理解为consumer group才是topic在逻辑上的订阅者。
每个consumer可以订阅多个topic。
每个consumer会保留它读取到某个partition的offset。而consumer 是通过zookeeper来保留offset的
三、Kafka 安装
单节点模式
#自己的笔记
[root@HadoopNode00 ~]# tar -zxvf kafka_2.11-0.11.0.0.tgz -C /home/kafka/
#自己的笔记
# 修改/home/kafka/kafka_2.11-0.11.0.0/config/server.properties配置文件
#id
#broker.id=0 如果是单节点的话，不用改
#日志存储目录
#log.dirs=/home/kafka/kafka_2.11-0.11.0.0/kafka-logs/
#zk地址
#zookeeper.connect=HadoopNode00:2181
#允许删除topic
#delete.topic.enable=true
# 配置kafak sever 的地址 --- kafka的默认端口号是9092
#listeners=PLAINTEXT://HadoopNode00:9092
#消息存储时长
#log.retention.hours=168 --- 7天
#超时时间
#zookeeper.connection.timeout.ms=6000
#自己的笔记
[root@HadoopNode00 ~]# cd /home/
[root@HadoopNode00 home]# ls
1.txt 2.txt findbugs flume hadoop hbase hive java kafka maven protobuf zk
[root@HadoopNode00 home]# cd kafka/
[root@HadoopNode00 kafka]# ls
kafka_2.11-0.11.0.0
[root@HadoopNode00 kafka]# cd kafka_2.11-0.11.0.0/
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ls
bin config kafka-logs libs LICENSE NOTICE site-docs
[root@HadoopNode00 kafka_2.11-0.11.0.0]# cd bin/
[root@HadoopNode00 bin]# ls
connect-distributed.sh kafka-consumer-offset-checker.sh kafka-replica-verification.sh kafka-verifiable-producer.sh
connect-standalone.sh kafka-consumer-perf-test.sh kafka-run-class.sh windows
kafka-acls.sh kafka-delete-records.sh kafka-server-start.sh zookeeper-security-migration.sh
kafka-broker-api-versions.sh kafka-mirror-maker.sh kafka-server-stop.sh zookeeper-server-start.sh
kafka-configs.sh kafka-preferred-replica-election.sh kafka-simple-consumer-shell.sh zookeeper-server-stop.sh
kafka-console-consumer.sh kafka-producer-perf-test.sh kafka-streams-application-reset.sh zookeeper-shell.sh
kafka-console-producer.sh kafka-reassign-partitions.sh kafka-topics.sh
kafka-consumer-groups.sh kafka-replay-log-producer.sh kafka-verifiable-consumer.sh
[root@HadoopNode00 bin]#
#自己的笔记
#首先要启动zk
[root@HadoopNode00 zookeeper-3.4.6]# ./bin/zkServer.sh start ./conf/zk.cfg
JMX enabled by default
Using config: ./conf/zk.cfg
Starting zookeeper ... STARTED
[root@HadoopNode00 zookeeper-3.4.6]# jps
122195 QuorumPeerMain #保证zk正常启动
122220 Jps
[root@HadoopNode00 zookeeper-3.4.6]#
#自己的笔记
#前台启动的所有日志信息 ---> 处于挂起状态
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-server-start.sh ./config/server.properties # 前台方式启动
[2019-11-12 10:43:49,206] INFO KafkaConfig values:
advertised.host.name = null
advertised.listeners = null
advertised.port = null
alter.config.policy.class.name = null
authorizer.class.name =
auto.create.topics.enable = true
auto.leader.rebalance.enable = true
background.threads = 10
broker.id = 0
broker.id.generation.enable = true
broker.rack = null
compression.type = producer
connections.max.idle.ms = 600000
controlled.shutdown.enable = true
controlled.shutdown.max.retries = 3
controlled.shutdown.retry.backoff.ms = 5000
controller.socket.timeout.ms = 30000
create.topic.policy.class.name = null
default.replication.factor = 1
delete.records.purgatory.purge.interval.requests = 1
delete.topic.enable = true
fetch.purgatory.purge.interval.requests = 1000
group.initial.rebalance.delay.ms = 0
group.max.session.timeout.ms = 300000
group.min.session.timeout.ms = 6000
host.name =
inter.broker.listener.name = null
inter.broker.protocol.version = 0.11.0-IV2
leader.imbalance.check.interval.seconds = 300
leader.imbalance.per.broker.percentage = 10
listener.security.protocol.map = SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,TRACE:TRACE,SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT
listeners = PLAINTEXT://HadoopNode00:9092
log.cleaner.backoff.ms = 15000
log.cleaner.dedupe.buffer.size = 134217728
log.cleaner.delete.retention.ms = 86400000
log.cleaner.enable = true
log.cleaner.io.buffer.load.factor = 0.9
log.cleaner.io.buffer.size = 524288
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
log.cleaner.min.cleanable.ratio = 0.5
log.cleaner.min.compaction.lag.ms = 0
log.cleaner.threads = 1
log.cleanup.policy = [delete]
log.dir = /tmp/kafka-logs
log.dirs = /home/kafka/kafka_2.11-0.11.0.0/kafka-logs/
log.flush.interval.messages = 9223372036854775807
log.flush.interval.ms = null
log.flush.offset.checkpoint.interval.ms = 60000
log.flush.scheduler.interval.ms = 9223372036854775807
log.flush.start.offset.checkpoint.interval.ms = 60000
log.index.interval.bytes = 4096
log.index.size.max.bytes = 10485760
log.message.format.version = 0.11.0-IV2
log.message.timestamp.difference.max.ms = 9223372036854775807
log.message.timestamp.type = CreateTime
log.preallocate = false
log.retention.bytes = -1
log.retention.check.interval.ms = 300000
log.retention.hours = 168
log.retention.minutes = null
log.retention.ms = null
log.roll.hours = 168
log.roll.jitter.hours = 0
log.roll.jitter.ms = null
log.roll.ms = null
log.segment.bytes = 1073741824
log.segment.delete.delay.ms = 60000
max.connections.per.ip = 2147483647
max.connections.per.ip.overrides =
message.max.bytes = 1000012
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
min.insync.replicas = 1
num.io.threads = 8
num.network.threads = 3
num.partitions = 1
num.recovery.threads.per.data.dir = 1
num.replica.fetchers = 1
offset.metadata.max.bytes = 4096
offsets.commit.required.acks = -1
offsets.commit.timeout.ms = 5000
offsets.load.buffer.size = 5242880
offsets.retention.check.interval.ms = 600000
offsets.retention.minutes = 1440
offsets.topic.compression.codec = 0
offsets.topic.num.partitions = 50
offsets.topic.replication.factor = 1
offsets.topic.segment.bytes = 104857600
port = 9092
principal.builder.class = class org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
producer.purgatory.purge.interval.requests = 1000
queued.max.requests = 500
quota.consumer.default = 9223372036854775807
quota.producer.default = 9223372036854775807
quota.window.num = 11
quota.window.size.seconds = 1
replica.fetch.backoff.ms = 1000
replica.fetch.max.bytes = 1048576
replica.fetch.min.bytes = 1
replica.fetch.response.max.bytes = 10485760
replica.fetch.wait.max.ms = 500
replica.high.watermark.checkpoint.interval.ms = 5000
replica.lag.time.max.ms = 10000
replica.socket.receive.buffer.bytes = 65536
replica.socket.timeout.ms = 30000
replication.quota.window.num = 11
replication.quota.window.size.seconds = 1
request.timeout.ms = 30000
reserved.broker.max.id = 1000
sasl.enabled.mechanisms = [GSSAPI]
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.principal.to.local.rules = [DEFAULT]
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism.inter.broker.protocol = GSSAPI
security.inter.broker.protocol = PLAINTEXT
socket.receive.buffer.bytes = 102400
socket.request.max.bytes = 104857600
socket.send.buffer.bytes = 102400
ssl.cipher.suites = null
ssl.client.auth = none
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.abort.timed.out.transaction.cleanup.interval.ms = 60000
transaction.max.timeout.ms = 900000
transaction.remove.expired.transaction.cleanup.interval.ms = 3600000
transaction.state.log.load.buffer.size = 5242880
transaction.state.log.min.isr = 1
transaction.state.log.num.partitions = 50
transaction.state.log.replication.factor = 1
transaction.state.log.segment.bytes = 104857600
transactional.id.expiration.ms = 604800000
unclean.leader.election.enable = false
zookeeper.connect = HadoopNode00:2181
zookeeper.connection.timeout.ms = 6000
zookeeper.session.timeout.ms = 6000
zookeeper.set.acl = false
zookeeper.sync.time.ms = 2000
(kafka.server.KafkaConfig)
[2019-11-12 10:43:49,553] INFO starting (kafka.server.KafkaServer)
[2019-11-12 10:43:49,563] INFO Connecting to zookeeper on HadoopNode00:2181 (kafka.server.KafkaServer)
[2019-11-12 10:43:49,623] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2019-11-12 10:43:49,752] INFO Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:host.name=HadoopNode00 (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:java.version=1.8.0_181 (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:java.home=/home/java/jdk1.8.0_181/jre (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:java.class.path=:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/aopalliance-repackaged-2.5.0-b05.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/argparse4j-0.7.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/commons-lang3-3.5.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/connect-api-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/connect-file-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/connect-json-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/connect-runtime-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/connect-transforms-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/guava-20.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/hk2-api-2.5.0-b05.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/hk2-locator-2.5.0-b05.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/hk2-utils-2.5.0-b05.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jackson-annotations-2.8.5.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jackson-core-2.8.5.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jackson-databind-2.8.5.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jackson-jaxrs-base-2.8.5.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jackson-jaxrs-json-provider-2.8.5.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jackson-module-jaxb-annotations-2.8.5.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/javassist-3.21.0-GA.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/javax.annotation-api-1.2.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/javax.inject-1.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/javax.inject-2.5.0-b05.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/javax.servlet-api-3.1.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jersey-client-2.24.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jersey-common-2.24.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jersey-container-servlet-2.24.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jersey-container-servlet-core-2.24.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jersey-guava-2.24.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jersey-media-jaxb-2.24.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jersey-server-2.24.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/jopt-simple-5.0.3.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/kafka_2.11-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/kafka_2.11-0.11.0.0-sources.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/kafka_2.11-0.11.0.0-test-sources.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/kafka-clients-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/kafka-log4j-appender-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/kafka-streams-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/kafka-streams-examples-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/kafka-tools-0.11.0.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/log4j-1.2.17.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/lz4-1.3.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/maven-artifact-3.5.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/metrics-core-2.2.0.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/plexus-utils-3.0.24.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/reflections-0.9.11.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/rocksdbjni-5.0.1.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/scala-library-2.11.11.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/slf4j-api-1.7.25.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/slf4j-log4j12-1.7.25.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/snappy-java-1.1.2.6.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/validation-api-1.1.0.Final.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/zkclient-0.10.jar:/home/kafka/kafka_2.11-0.11.0.0/bin/../libs/zookeeper-3.4.10.jar (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,752] INFO Client environment:os.version=2.6.32-754.el6.x86_64 (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,753] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,753] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,753] INFO Client environment:user.dir=/home/kafka/kafka_2.11-0.11.0.0 (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,830] INFO Initiating client connection, connectString=HadoopNode00:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@158da8e (org.apache.zookeeper.ZooKeeper)
[2019-11-12 10:43:49,842] INFO Opening socket connection to server HadoopNode00/192.168.89.101:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2019-11-12 10:43:49,843] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2019-11-12 10:43:49,883] INFO Socket connection established to HadoopNode00/192.168.89.101:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2019-11-12 10:43:49,959] INFO Session establishment complete on server HadoopNode00/192.168.89.101:2181, sessionid = 0x16e5d7de4470000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2019-11-12 10:43:50,014] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2019-11-12 10:43:50,287] INFO Cluster ID = xG0fvOEqS9awea6U8ZsOQg (kafka.server.KafkaServer)
[2019-11-12 10:43:50,298] WARN No meta.properties file under dir /home/kafka/kafka_2.11-0.11.0.0/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2019-11-12 10:43:50,519] INFO [ThrottledRequestReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-11-12 10:43:50,633] INFO [ThrottledRequestReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-11-12 10:43:50,897] INFO [ThrottledRequestReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2019-11-12 10:43:50,976] INFO Loading logs. (kafka.log.LogManager)
[2019-11-12 10:43:50,984] INFO Logs loading complete in 8 ms. (kafka.log.LogManager)
[2019-11-12 10:43:54,296] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2019-11-12 10:43:54,298] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2019-11-12 10:43:54,399] INFO Awaiting socket connections on HadoopNode00:9092. (kafka.network.Acceptor)
[2019-11-12 10:43:54,407] INFO [Socket Server on Broker 0], Started 1 acceptor threads (kafka.network.SocketServer)
[2019-11-12 10:43:54,473] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-11-12 10:43:54,473] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-11-12 10:43:54,473] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-11-12 10:43:54,677] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-11-12 10:43:54,713] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-11-12 10:43:54,717] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-11-12 10:43:54,723] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2019-11-12 10:43:54,738] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2019-11-12 10:43:54,907] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2019-11-12 10:43:54,922] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2019-11-12 10:43:54,936] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 10 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-11-12 10:43:54,959] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2019-11-12 10:43:55,077] INFO [Transaction Coordinator 0]: Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-11-12 10:43:55,084] INFO [Transaction Coordinator 0]: Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-11-12 10:43:55,089] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2019-11-12 10:43:55,377] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2019-11-12 10:43:55,463] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2019-11-12 10:43:55,544] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2019-11-12 10:43:55,545] INFO Registered broker 0 at path /brokers/ids/0 with addresses: EndPoint(HadoopNode00,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2019-11-12 10:43:55,545] WARN No meta.properties file under dir /home/kafka/kafka_2.11-0.11.0.0/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2019-11-12 10:43:55,782] INFO Kafka version : 0.11.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2019-11-12 10:43:55,783] INFO Kafka commitId : cb8625948210849f (org.apache.kafka.common.utils.AppInfoParser)
[2019-11-12 10:43:55,784] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
#自己的笔记
#查看启动的状态
[root@HadoopNode00 zookeeper-3.4.6]# jps
122625 Kafka #kafka的进程
122195 QuorumPeerMain
123544 Jps
[root@HadoopNode00 zookeeper-3.4.6]#
#自己的笔记
#关闭kafka
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-server-stop.sh
#自己的笔记
#查看kafka是否关闭成功
[root@HadoopNode00 kafka_2.11-0.11.0.0]# jps
122625 Kafka #注意
122195 QuorumPeerMain
124345 Jps
[root@HadoopNode00 kafka_2.11-0.11.0.0]# jps
122195 QuorumPeerMain
124379 Jps
#自己的笔记
#复习zk
[root@HadoopNode00 ~]# cd /home/
[root@HadoopNode00 home]# ls
1.txt 2.txt findbugs flume hadoop hbase hive java kafka maven protobuf zk
[root@HadoopNode00 home]# cd zk/
[root@HadoopNode00 zk]# ls
data data01 data02 data03 zookeeper-3.4.6
[root@HadoopNode00 zk]# cd zookeeper-3.4.6/
[root@HadoopNode00 zookeeper-3.4.6]# ls
bin conf docs lib README_packaging.txt src zookeeper-3.4.6.jar.md5
build.xml contrib ivysettings.xml LICENSE.txt README.txt zookeeper-3.4.6.jar zookeeper-3.4.6.jar.sha1
CHANGES.txt dist-maven ivy.xml NOTICE.txt recipes zookeeper-3.4.6.jar.asc zookeeper.out
[root@HadoopNode00 zookeeper-3.4.6]# cd bin/
[root@HadoopNode00 bin]# ls
README.txt zkCleanup.sh zkCli.cmd zkCli.sh zkEnv.cmd zkEnv.sh zkServer.cmd zkServer.sh
[root@HadoopNode00 bin]# cd ..
#自己的笔记
[root@HadoopNode00 zookeeper-3.4.6]# ./bin/zkCli.sh Connecting to localhost:2181
2019-11-12 10:50:59,625 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2019-11-12 10:50:59,628 [myid:] - INFO [main:Environment@100] - Client environment:host.name=HadoopNode00
2019-11-12 10:50:59,628 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_181
2019-11-12 10:50:59,630 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/home/java/jdk1.8.0_181/jre
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/home/zk/zookeeper-3.4.6/bin/../build/classes:/home/zk/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/zk/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/zk/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/zk/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/zk/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/zk/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/zk/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/zk/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/zk/zookeeper-3.4.6/bin/../conf:
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-754.el6.x86_64
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root
2019-11-12 10:50:59,631 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/home/zk/zookeeper-3.4.6
2019-11-12 10:50:59,634 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@506c589e
Welcome to ZooKeeper!
2019-11-12 10:50:59,701 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2019-11-12 10:50:59,912 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@852] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2019-11-12 10:50:59,919 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x16e5d7de4470001, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
#自己的笔记
[zk: localhost:2181(CONNECTED) 0] ls /
[cluster, controller_epoch, brokers, zookeeper, 123123, admin, isr_change_notification, #consumers,#此处 latest_producer_id_block, config, hbase]
#自己的笔记
[zk: localhost:2181(CONNECTED) 1] ls cluster
Command failed: java.lang.IllegalArgumentException: Path must start with / character
[zk: localhost:2181(CONNECTED) 2] ls /cluster
[id]
#自己的笔记
[zk: localhost:2181(CONNECTED) 3] ls /cluster.id
Node does not exist: /cluster.id
[zk: localhost:2181(CONNECTED) 4] ls /cluster/id
[]
[zk: localhost:2181(CONNECTED) 5] get /cluster/id
{"version":"1","id":"xG0fvOEqS9awea6U8ZsOQg"}
cZxid = 0x364
ctime = Tue Nov 12 10:43:50 CST 2019
mZxid = 0x364
mtime = Tue Nov 12 10:43:50 CST 2019
pZxid = 0x364
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 45
numChildren = 0
#自己的笔记
[zk: localhost:2181(CONNECTED) 6] get /cluster
null
cZxid = 0x363
ctime = Tue Nov 12 10:43:50 CST 2019
mZxid = 0x363
mtime = Tue Nov 12 10:43:50 CST 2019
pZxid = 0x364
cversion = 1
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 1
#自己的笔记
[zk: localhost:2181(CONNECTED) 7] ls /consumers
[]
#自己的笔记
[zk: localhost:2181(CONNECTED) 8] ls /brokers
[ids, topics, seqid]
#自己的笔记
[zk: localhost:2181(CONNECTED) 9] ls /brokers/ids
[]
[zk: localhost:2181(CONNECTED) 10] get /brokersid
Node does not exist: /brokersid
[zk: localhost:2181(CONNECTED) 11] get /brokers/ids
null
cZxid = 0x355
ctime = Tue Nov 12 10:43:50 CST 2019
mZxid = 0x355
mtime = Tue Nov 12 10:43:50 CST 2019
pZxid = 0x36d
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
#自己的笔记
[zk: localhost:2181(CONNECTED) 14] ls /brokers
[ids, topics, seqid]
[zk: localhost:2181(CONNECTED) 15] ls /brokers/ids
[]
[zk: localhost:2181(CONNECTED) 16] ls /brokers/topics
[]
[zk: localhost:2181(CONNECTED) 17] ls /brokers/seqid
[]
[zk: localhost:2181(CONNECTED) 18] get /brokers/
ids topics seqid
[zk: localhost:2181(CONNECTED) 18] get /brokers/ids
null
cZxid = 0x355
ctime = Tue Nov 12 10:43:50 CST 2019
mZxid = 0x355
mtime = Tue Nov 12 10:43:50 CST 2019
pZxid = 0x36d
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
[zk: localhost:2181(CONNECTED) 19] get /brokers/topics
null
cZxid = 0x356
ctime = Tue Nov 12 10:43:50 CST 2019
mZxid = 0x356
mtime = Tue Nov 12 10:43:50 CST 2019
pZxid = 0x356
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
[zk: localhost:2181(CONNECTED) 20] get /brokers/seqid
null
cZxid = 0x35f
ctime = Tue Nov 12 10:43:50 CST 2019
mZxid = 0x35f
mtime = Tue Nov 12 10:43:50 CST 2019
pZxid = 0x35f
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 0
#自己的笔记
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-server-start.sh -daemon ./config/server.properties # 后台台方式启动
#自己的笔记
[root@HadoopNode00 kafka_2.11-0.11.0.0]# jps
122195 QuorumPeerMain
125556 ZooKeeperMain # ？？？
28253 Jps
28221 Kafka #注意此处
[root@HadoopNode00 ~]# mkdir /home/kafka
[root@HadoopNode00 ~]# tar -zxvf kafka_2.11-0.11.0.0.tgz -C /home/kafka/
# /home/kafka/kafka_2.11-0.11.0.0/config/server.properties
# Switch to enable topic deletion or not, default value is false 允许删除topic
delete.topic.enable=true
# 配置kafak sever 的地址
listeners=PLAINTEXT://HadoopNode00:9092
# A comma seperated list of directories under which to store log files 日志存储目录
log.dirs=/home/kafka/kafka_2.11-0.11.0.0/kafka-logs/
#zk地址
zookeeper.connect=HadoopNode00:2181
# Timeout in ms for connecting to zookeeper 超时时间
zookeeper.connection.timeout.ms=6000
# The minimum age of a log file to be eligible for deletion due to age 消息存储时长
log.retention.hours=168
启动
zk 正常启动
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-server-start.sh ./config/server.properties # 前台方式启动
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-server-start.sh -daemon ./config/server.properties
# 后台台方式启动
关闭
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-server-stop.sh
集群模式
#自己的笔记
#节点一:
[root@HadoopNode01 ~]# tar -zxvf kafka_2.11-0.11.0.0.tgz -C /home/kafka/
#broker.id=0
#允许删除topic
#delete.topic.enable=true
#配置kafka-server的地址
#listeners=PLAINTEXT://HadoopNode01:9092
#日志存储目录
#log.dirs=/home/kafka/kafka_2.11-0.11.0.0/kafka-logs/
#要创建kafka-logs文件夹
#zk地址设置为集群模式
#zk地址
#zookeeper.connect=HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181
#自己的笔记
#编写脚本文件
[root@HadoopNode01 zookeeper-3.4.6]# vi zkconn.sh #位置搞错了吧
[root@HadoopNode01 zookeeper-3.4.6]# more zkconn.sh
#!/usr/bin/env bash
echo "连接zk"
echo " ————————————————————"
echo "| author：MaBingYang |"
echo "| company：Baizhi |"
echo " ————————————————————"
/home/zk/zookeeper-3.4.6/bin/zkServer.sh start /home/zk/zookeeper-3.4.6/conf/zk.cfg
sleep 1
[root@HadoopNode01 zookeeper-3.4.6]#
#自己的笔记
#shell脚本测试
[root@HadoopNode01 ~]# sh zk
sh: zk: No such file or directory
[root@HadoopNode01 ~]# sh zkconn.sh
连接zk
————————————————————
| author：MaBingYang |
| company：Baizhi |
————————————————————
JMX enabled by default
Using config: /home/zk/zookeeper-3.4.6/conf/zk.cfg
Starting zookeeper ... jSTARTED
ps
[root@HadoopNode01 ~]# jps
10039 QuorumPeerMain
10063 Jps
[root@HadoopNode01 ~]#
#自己的笔记
#编写kafka脚本文件
[root@HadoopNode02 ~]# vi startkafka.sh
[root@HadoopNode02 ~]# more startkafka.sh
#!/usr/bin/env bash
echo "startkafka"
echo " ————————————————————"
echo "| author：MaBingYang |"
echo "| company：Baizhi |"
echo " ————————————————————"
/home/kafka/kafka_2.11-0.11.0.0/bin/kafka-server-start.sh -daemon /home/kafka/kafka_2.11-0.11.0.0/config/server.properties
sleep 1
#自己的笔记
#测试脚本文件
[root@HadoopNode02 ~]# sh startkafka.sh
startkafka
————————————————————
| author：MaBingYang |
| company：Baizhi |
————————————————————
[root@HadoopNode02 ~]# jps
5890 Kafka #测试成功
4356 QuorumPeerMain
5967 Jps
[root@HadoopNode02 ~]#
#节点二:
[root@HadoopNode01 ~]# tar -zxvf kafka_2.11-0.11.0.0.tgz -C /home/kafka/
#broker.id=1
#允许删除topic
#delete.topic.enable=true
#配置kafka-server的地址
#listeners=PLAINTEXT://HadoopNode02:9092
#日志存储目录
#log.dirs=/home/kafka/kafka_2.11-0.11.0.0/kafka-logs/
#要创建kafka-logs文件夹
#zk地址设置为集群模式
#zk地址
#zookeeper.connect=HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181
#自己的笔记
#编写zk脚本文件
[root@HadoopNode01 zookeeper-3.4.6]# vi zkconn.sh #位置搞错了吧
[root@HadoopNode01 zookeeper-3.4.6]# more zkconn.sh
#!/usr/bin/env bash
echo "连接zk"
echo " ————————————————————"
echo "| author：MaBingYang |"
echo "| company：Baizhi |"
echo " ————————————————————"
/home/zk/zookeeper-3.4.6/bin/zkServer.sh start /home/zk/zookeeper-3.4.6/conf/zk.cfg
sleep 1
[root@HadoopNode01 zookeeper-3.4.6]#
#自己的笔记
#shell脚本测试
[root@HadoopNode02 ~]# sh zkconn.sh
连接zk
————————————————————
| author：MaBingYang |
| company：Baizhi |
————————————————————
JMX enabled by default
Using config: /home/zk/zookeeper-3.4.6/conf/zk.cfg
Starting zookeeper ... jSTARTED
ps
[root@HadoopNode02 ~]# jps
4356 QuorumPeerMain
4382 Jps
[root@HadoopNode02 ~]#
#自己的笔记
#编写shell脚本文件
[root@HadoopNode01 ~]# vi startkafka.sh
[root@HadoopNode01 ~]# more startkafka.sh
#!/usr/bin/env bash
echo "连接zk"
echo " ————————————————————"
echo "| author：MaBingYang |"
echo "| company：Baizhi |"
echo " ————————————————————"
/home/kafka/kafka_2.11-0.11.0.0/bin/kafka-server-start.sh -daemon /home/kafka/kafka_2.11-0.11.0.0/config/server.properties
sleep 1
[root@HadoopNode01 ~]# vi startkafka.sh
[root@HadoopNode01 ~]# more startkafka.sh
#!/usr/bin/env bash
echo "startkafka"
echo " ————————————————————"
echo "| author：MaBingYang |"
echo "| company：Baizhi |"
echo " ————————————————————"
/home/kafka/kafka_2.11-0.11.0.0/bin/kafka-server-start.sh -daemon /home/kafka/kafka_2.11-0.11.0.0/config/server.properties
sleep 1
[root@HadoopNode01 ~]#
#自己的笔记
#测试脚本文件
[root@HadoopNode01 ~]# sh startkafka.sh
startkafka
————————————————————
| author：MaBingYang |
| company：Baizhi |
————————————————————
[root@HadoopNode01 ~]# jps
10039 QuorumPeerMain
17322 Jps
17243 Kafka #测试成功
[root@HadoopNode01 ~]#
#节点三:
[root@HadoopNode01 ~]# tar -zxvf kafka_2.11-0.11.0.0.tgz -C /home/kafka/
#broker.id=2
#允许删除topic
#delete.topic.enable=true
#配置kafka-server的地址
#listeners=PLAINTEXT://HadoopNode03:9092
#日志存储目录
#log.dirs=/home/kafka/kafka_2.11-0.11.0.0/kafka-logs/
#要创建kafka-logs文件夹
#zk地址设置为集群模式
#zk地址
#zookeeper.connect=HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181
#自己的笔记
#编写脚本文件
[root@HadoopNode01 zookeeper-3.4.6]# vi zkconn.sh #位置搞错了吧
[root@HadoopNode01 zookeeper-3.4.6]# more zkconn.sh
#!/usr/bin/env bash
echo "连接zk"
echo " ————————————————————"
echo "| author：MaBingYang |"
echo "| company：Baizhi |"
echo " ————————————————————"
/home/zk/zookeeper-3.4.6/bin/zkServer.sh start /home/zk/zookeeper-3.4.6/conf/zk.cfg
sleep 1
[root@HadoopNode01 zookeeper-3.4.6]#
#自己的笔记
#shell脚本测试
[root@HadoopNode03 ~]# sh zkconn.sh
连接zk
————————————————————
| author：MaBingYang |
| company：Baizhi |
————————————————————
JMX enabled by default
Using config: /home/zk/zookeeper-3.4.6/conf/zk.cfg
Starting zookeeper ... jSTARTED
ps
[root@HadoopNode03 ~]# jps
4481 -- main class information unavailable
4505 Jps
[root@HadoopNode03 ~]# jps
4481 -- main class information unavailable
4581 Jps
[root@HadoopNode03 ~]# jps
4481 QuorumPeerMain
4633 Jps
[root@HadoopNode03 ~]#
#自己的笔记
#启动节点一
[root@HadoopNode01 kafka_2.11-0.11.0.0]# /home/kafka/kafka_2.11-0.11.0.0/bin/kafka-server-start.sh /home/kafka/kafka_2.11-0.11.0.0/config/server.properties
#自己的笔记
#启动成功后的部分日志信息
[2019-11-13 11:45:48,419] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2019-11-13 11:45:48,494] INFO [Transaction Coordinator 0]: Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-11-13 11:45:48,505] INFO [Transaction Coordinator 0]: Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-11-13 11:45:48,515] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2019-11-13 11:45:48,618] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2019-11-13 11:45:48,671] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2019-11-13 11:45:48,682] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2019-11-13 11:45:48,687] INFO Registered broker 0 at path /brokers/ids/0 with addresses: EndPoint(HadoopNode01,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.utils.ZkUtils)
[2019-11-13 11:45:48,688] WARN No meta.properties file under dir /home/kafka/kafka_2.11-0.11.0.0/kafka-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2019-11-13 11:45:49,437] INFO Kafka version : 0.11.0.0 (org.apache.kafka.common.utils.AppInfoParser)
[2019-11-13 11:45:49,437] INFO Kafka commitId : cb8625948210849f (org.apache.kafka.common.utils.AppInfoParser)
[2019-11-13 11:45:49,440] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
#自己的笔记
[root@HadoopNode03 ~]# vi startkafka.sh
[root@HadoopNode03 ~]# more startkafka.sh
#!/usr/bin/env bash
echo "startkafka"
echo " ————————————————————"
echo "| author：MaBingYang |"
echo "| company：Baizhi |"
echo " ————————————————————"
/home/kafka/kafka_2.11-0.11.0.0/bin/kafka-server-start.sh -daemon /home/kafka/kafka_2.11-0.11.0.0/config/server.properties
sleep 1
#自己的笔记
#测试编写的脚本文件
[root@HadoopNode03 ~]# sh startkafka.sh
startkafka
————————————————————
| author：MaBingYang |
| company：Baizhi |
————————————————————
[root@HadoopNode03 ~]# jps
4481 QuorumPeerMain
6418 Jps
6356 Kafka #测试成功
[root@HadoopNode03 ~]#
# /home/kafka/kafka_2.11-0.11.0.0/config/server.properties
# The id of the broker. This must be set to a unique integer for each broker. # broker id 必须不一样
broker.id=[0,1,2]
# Switch to enable topic deletion or not, default value is false 允许删除topic
delete.topic.enable=true
# 配置kafak sever 的地址
listeners=PLAINTEXT://[HadoopNode01,HadoopNode02，HadoopNode03]:9092
# A comma seperated list of directories under which to store log files 日志存储目录
log.dirs=/home/kafka/kafka_2.11-0.11.0.0/kafka-logs/
#zk地址
zookeeper.connect=HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181
# Timeout in ms for connecting to zookeeper 超时时间
zookeeper.connection.timeout.ms=6000
# The minimum age of a log file to be eligible for deletion due to age 消息存储时长
log.retention.hours=168
四、使用
验证启动
此时如果使用的是单节点 创建分区副本数量一般就指定为1，因为指定为大于1 的情况后，会发生报错，因为Broker的数量不够 或者说分区副本数量要小于等等于Broker的数量
#单节点下命令的测试
#自己的笔记
#创建主题
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookee per HadoopNode00:2181 --create --topic t01 --partitions 3 -replication- factor 3
Error while executing topic command : replication factor: 3 larger than available brokers: 1
[2019-11-12 13:56:17,944] ERROR org.apache.kafka.common.errors.InvalidR eplicationFactorException: replication factor: 3 larger than available brokers: 1
(kafka.admin.TopicCommand$)
#自己的笔记
#创建的指令是要连接 zk：2181
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode00:2181 --create --topic t01 --partitions 3 -replication-factor 1
Created topic "t01".
[root@HadoopNode00 kafka_2.11-0.11.0.0]#
#自己的笔记
#启动生产者 ---- kafka提供的一个Demo
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-console-producer.sh -broker-list HadoopNode00:9092 --topic t01
1
2
3
4
5
6
7
8
9>1
2
3
4
>>>>>>>>>>>> #正是开始发送消息 ---- 观察消费者的消费情况
>1
>2
>3
>4
>5
>6
>7
>8
>9
>
>0000
>0000
#自己的笔记
#启动消费者的指令是要连接 bootstrap-server kafka:9092
#启动消费者 ---- kafka提供的一个Demo
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-console-consumer.sh --bootstrap-server HadoopNode00:9092 --topic t01
#处于挂起状态
2
5
8
3
1
4
7
2
3
6
91
4
1
2
4
3
5
7
6
8
9
0000
0000
#集群模式下命令的测试
#自己的笔记
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --create --topic t01 --partitions 3 -replication-factor 3
Created topic "t01".
#自己的笔记
#先启动消费者 --- 处于挂起转态
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-console-consumer.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t01
#节点一下的消费者消费的数据
1
1
2
3
4
5
6
7
8
9
90
#自己的笔记
#节点三下启动启动生产者 --- 节点二下启动生产者有BUG
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-console-producer.sh --broker-list HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t01
>1
>2
>3
>4
>5
>6
>7
>8
>9
>90
>
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --create --topic t01 --partitions 3 -replication-factor 3
Created topic "t01".
# 消费者
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-console-consumer.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t01
# 生产者
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-console-producer.sh --broker-list HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t01
添加Topic
#自己的笔记
#集群模式下
#添加topic02
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --create --topic t02 --partitions 3 -replication-factor 3
Created topic "t02".
[root@HadoopNode01 kafka_2.11-0.11.0.0]#
#自己的笔记
#集群模式下
#测试添加的topic02 --- 展示所有的topic
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --list
__consumer_offsets #默认的topic --- 记录偏移量的。
t01
t02
[root@HadoopNode01 kafka_2.11-0.11.0.0]#
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --create --topic t01 --partitions 3 -replication-factor 3
Created topic "t01".
查看Topic 详情
#自己的笔记
#单机模式下: 3分区-1副本
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode00:2181 --describe --topic t01
Topic:t01 PartitionCount:3 ReplicationFactor:1 Configs:
Topic: t01 Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: t01 Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: t01 Partition: 2 Leader: 0 Replicas: 0 Isr: 0
#自己的笔记
#集群模式下: 3分区-3副本
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --describe --topic t01
Topic:t01 PartitionCount:3 ReplicationFactor:3 Configs:
Topic: t01 Partition: 0 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: t01 Partition: 1 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
Topic: t01 Partition: 2 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode00:2181 --describe --topic t01 # 单节点
Topic:t01 PartitionCount:3 ReplicationFactor:1 Configs:
Topic: t01 Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: t01 Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: t01 Partition: 2 Leader: 0 Replicas: 0 Isr: 0
[root@HadoopNode00 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --describe --topic t01 # 集群
删除Topic
#自己的笔记
#集群模式下:
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --delete --topic t04
Topic t04 is marked for deletion. --- 没有设置为true的是标记删除(不是真正的删除)
Note: This will have no impact if delete.topic.enable is not set to true.
修改分区信息
#自己的笔记
#集群模式下:
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --alter --topic t01 --partitions 4
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[root@HadoopNode03 kafka_2.11-
#自己的笔记
#集群模式下:
#修改的测是 4分区-3副本
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --describe --topic t01
Topic:t01 PartitionCount:4 ReplicationFactor:3 Configs:
Topic: t01 Partition: 0 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
Topic: t01 Partition: 1 Leader: 0 Replicas: 0,2,1 Isr: 0,2,1
Topic: t01 Partition: 2 Leader: 1 Replicas: 1,0,2 Isr: 1,0,2
Topic: t01 Partition: 3 Leader: 2 Replicas: 2,1,0 Isr: 2,1,0
#自己的笔记
#集群模式下：
#创建topic03 3分区-2副本
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --create --topic t03 --partitions 3 -replication-factor 2
Created topic "t03".
#自己的笔记
#集群模式下；
#展示分区信息
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --describe --topic t03
Topic:t03 PartitionCount:3 ReplicationFactor:2 Configs:
Topic: t03 Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0
Topic: t03 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: t03 Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2
#自己的笔记
#集群模式下: 2分区-2副本 --- 两个分区就有两个topic
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --create --topic t04 --partitions 2 -replication-factor 2
Created topic "t04".
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --describe --topic t04
Topic:t04 PartitionCount:2 ReplicationFactor:2 Configs:
Topic: t04 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: t04 Partition: 1 Leader: 2 Replicas: 2,0 Isr: 2,0
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --alter --topic t01 --partitions 4
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
显示所有Topic
#自己的笔记
#集群模式下:
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --list
__consumer_offsets
t01
t04
#自己的笔记
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-console-producer.sh --broker-list HadoopNode01:9092,HadoopNode02:9092,HadoopNode03
>1
>2
>3
>4
>5
>6
>7
>8
>9
>
#自己的笔记
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-console-consumer.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,Hadoop Node03:9092 --topic t01 -from-beginning
4
8
1
4
7
90
3
7
3
6
9
2
6
2
5
8
1
5
9
分区内先进先出
#自己的笔记
#测试数据
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --create --topic t06 --partitions 3 -replication-factor 3
Created topic "t06".
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-topics.sh --zookeeper HadoopNode01:2181,HadoopNode02:2181,HadoopNode03:2181 --describe --topic t06
Topic:t06 PartitionCount:3 ReplicationFactor:3 Configs:
Topic: t06 Partition: 0 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: t06 Partition: 1 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: t06 Partition: 2 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
[root@HadoopNode03 kafka_2.11-0.11.0.0]#
#自己的笔记
[root@HadoopNode03 kafka_2.11-0.11.0.0]# ./bin/kafka-console-producer.sh --broker-list HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t06
>1
>2
>3
>4
>5
>6
>7
>8
>9
>
#自己的笔记
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-console-consumer.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t06 -from-beginning
3
6
9
2
5
8
1
4
7
由此可见，kafka虽然也是FIFO，但是这个范围就是在分区内。 另外消费者在消费消息的时候，不确定是先消费那个分区。
删除某一则消息
这种操作极少，删除已经发布的一则或者多则消息
#自己的笔记
[root@HadoopNode02 ~]# vi del.json
[root@HadoopNode02 ~]# more del.json
{"partitions":
[
{"topic":"t04","partition":1,"offset":3}
]
}
#自己的笔记
#自己的笔记
[root@HadoopNode01 ~]# vi del.json
[root@HadoopNode01 ~]# cat del.json
{"partitions":
[
{"topic":"t04","partition":1,"offset":3}
]
}
[root@HadoopNode01 ~]#
#自己的笔记
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-delete-records.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --offset-json-file /root/del.json
Executing records delete operation
Records delete operation completed:
partition: t06-1 low_watermark: 3
#自己的笔记
#对比以下的数据
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-console-consumer.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t06 -from-beginning
3
6
9
2
5
8
1
4
7
^CProcessed a total of 9 messages
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-console-consumer.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t06 -from-beginning
3
6
9
2
5
8
#自己的笔记
[root@HadoopNode01 ~]# vi del.json
[root@HadoopNode01 ~]# cat del.json
{"partitions":
[
{"topic":"t06","partition":1,"offset":1}
]
}
#自己的笔记
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-delete-records.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --offset-json-file /root/del.json
Executing records delete operation
Records delete operation completed:
partition: t06-1 low_watermark: 3
#自己的笔记
#需要修改分区
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-console-consumer.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t06 -from-beginning
3
6
9
2
5
8
#自己的笔记
[root@HadoopNode01 ~]# vi del.json
[root@HadoopNode01 ~]# cat del.json
{"partitions":
[
{"topic":"t06","partition":0,"offset":1}
]
}
#自己的笔记
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-delete-records.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --offset-json-file /root/del.json
Executing records delete operation
Records delete operation completed:
partition: t06-0 low_watermark: 1
#自己的笔记
[root@HadoopNode01 kafka_2.11-0.11.0.0]# ./bin/kafka-console-consumer.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --topic t06 -from-beginning
3
6
9
5
8
[root@HadoopNode02 kafka_2.11-0.11.0.0]# ./bin/kafka-delete-records.sh --bootstrap-server HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092 --offset-json-file /root/del.json
{"partitions":
[
{"topic":"t04","partition":1,"offset":3}
]
}
删除t04中分区1的3以前的所有的数据
四、偏移量
#自己的笔记
#record--- 当做消息即可
Topic以日志分区形式存储,分区中每一则Record都有个offset编号用于标示record的顺序.所有存储在kafka中record是允许用户重复消费的。
#自己的笔记
kafka通过`log.retention.hours`控制Record存活时间. #168小时(7天)
#自己的笔记
kafka服务端只负责存储topic日志数据,kafka消费端独自维护一套分区信息和offset偏移量,一旦消费完成后消费端会自动提交消费分区的offset信息.下次在开始消费的时候只需要从上一次offset开始即可
五、Java API
创建Topic
@Test
public void create() {
Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092");
AdminClient adminClient = KafkaAdminClient.create(properties);
List<NewTopic> topic01 = Arrays.asList(new NewTopic("topic01", 3, (short) 3));
adminClient.createTopics(topic01);
adminClient.close();
}
删除Topic
@Test
public void del() {
Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092");
AdminClient adminClient = KafkaAdminClient.create(properties);
List<String> topic01 = Arrays.asList("topic01");
adminClient.deleteTopics(topic01);
adminClient.close();
}
列出Topic
@Test
public void list() throws Exception {
Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092");
AdminClient adminClient = KafkaAdminClient.create(properties);
adminClient.listTopics().names().get().forEach((name) -> {
System.out.println(name);
});
}
描述Topic
@Test
public void describe() throws Exception {
Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092");
AdminClient adminClient = KafkaAdminClient.create(properties);
List<String> topic01 = Arrays.asList("t01");
DescribeTopicsResult result = adminClient.describeTopics(topic01);
/*KafkaFuture<Map<String, TopicDescription>> all = result.all();
Map<String, TopicDescription> stringTopicDescriptionMap = all.get();
Set<String> strings = stringTopicDescriptionMap.keySet();
for (String string : strings) {
TopicDescription topicDescription = stringTopicDescriptionMap.get(string);
System.out.println(topicDescription.toString());
}*/
result.all().get().forEach((key, value) -> {
System.out.println(key + "---" + value);
});
}
发布Topic
@Test
public void publish() {
Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092");
/*
* 对key记性序列化的操作
* */
properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
/*
* 对value记性序列化的操作
* */
properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
/*
* 默认的分区操作
* */
properties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, DefaultPartitioner.class);
KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties);
for (int i = 0; i < 10; i++) {
ProducerRecord<String, String> producerRecord = new ProducerRecord<>("topic01", UUID.randomUUID().toString(), "this is zs"+i);
kafkaProducer.send(producerRecord);
}
kafkaProducer.flush();
kafkaProducer.close();
}
订阅Topic
在使用.subscribe订阅某个topic的时候，必须制定消费者组的ID，这样kafka才能实现负载均衡，才能实现组内均分和组件广播的策略。
@Test
public void subscribe() {
Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092");
/*
* key的反序列化处理
* */
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
/*
* value的反序列化处理
* */
properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
/*
* 指定组名
* */
properties.put(ConsumerConfig.GROUP_ID_CONFIG, "G1");
/*
* 创建KafkaConsume 对象 订阅某个topic
* */
KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<>(properties);
/*
* 订阅topic01
* */
kafkaConsumer.subscribe(Arrays.asList("topic01"));
/*
* 通过kafkaConsumer.poll 获取记录
* */
while (true) {
ConsumerRecords<String, String> consumerRecords = kafkaConsumer.poll(1000);
for (ConsumerRecord<String, String> record : consumerRecords) {
String key = record.key();
String value = record.value();
long offset = record.offset();
int partition = record.partition();
long timestamp = record.timestamp();
System.out.println("key:" + key + " value:" + value + " 偏移量:" + offset + " 分区：" + partition + " 时间戳：" + timestamp);
}
}
}
对象传输
@Test
public void subscribe() {
Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092");
/*
* key的反序列化处理
* */
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
/*
* value的反序列化处理
* */
properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
/*
* 指定组名
* */
properties.put(ConsumerConfig.GROUP_ID_CONFIG, "G1");
/*
* 创建KafkaConsume 对象 订阅某个topic
* */
KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<>(properties);
/*
* 订阅topic01
* */
kafkaConsumer.subscribe(Arrays.asList("topic01"));
/*
* 通过kafkaConsumer.poll 获取记录
* */
while (true) {
ConsumerRecords<String, String> consumerRecords = kafkaConsumer.poll(1000);
for (ConsumerRecord<String, String> record : consumerRecords) {
String key = record.key();
String value = record.value();
long offset = record.offset();
int partition = record.partition();
long timestamp = record.timestamp();
System.out.println("key:" + key + " value:" + value + " 偏移量:" + offset + " 分区：" + partition + " 时间戳：" + timestamp);
}
}
}
package com.baizhi.test01;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.internals.DefaultPartitioner;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;
import java.util.UUID;
public class KafkaUserProducer {
public static void main(String[] args) {
Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092");
/*
* 对key记性序列化的操作
* */
properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
/*
* 对value记性序列化的操作
* */
properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ObjectSerializer.class);
/*
* 默认的分区操作
* */
properties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, DefaultPartitioner.class);
KafkaProducer<String, User> kafkaProducer = new KafkaProducer<>(properties);
for (int i = 0; i < 10; i++) {
ProducerRecord<String, User> producerRecord = new ProducerRecord<>("topic03", UUID.randomUUID().toString(), new User(i + "", "zs" + i));
kafkaProducer.send(producerRecord);
}
kafkaProducer.flush();
kafkaProducer.close();
}
}
package com.baizhi.test01;
import org.apache.commons.lang3.SerializationUtils;
import org.apache.kafka.common.serialization.Deserializer;
import java.util.Map;
public class ObjectDeserializer implements Deserializer<Object> {
@Override
public void configure(Map<String, ?> map, boolean b) {
}
/*
* 反序列化 解码
* */
@Override
public Object deserialize(String s, byte[] bytes) {
return SerializationUtils.deserialize(bytes);
}
@Override
public void close() {
}
}
package com.baizhi.test01;
import org.apache.commons.lang3.SerializationUtils;
import org.apache.kafka.common.serialization.Serializer;
import java.io.Serializable;
import java.util.Map;
public class ObjectSerializer implements Serializer<Object> {
@Override
public void configure(Map<String, ?> map, boolean b) {
}
/*
* 序列化 编码
* */
@Override
public byte[] serialize(String s, Object o) {
return SerializationUtils.serialize((Serializable) o);
}
@Override
public void close() {
}
}
package com.baizhi.test01;
import java.io.Serializable;
public class User implements Serializable {
private String id;
private String name;
public User() {
}
public User(String id, String name) {
this.id = id;
this.name = name;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
@Override
public String toString() {
return "User{" +
"id='" + id + '\'' +
", name='" + name + '\'' +
'}';
}
}
偏移量控制
#自己的笔记 默认用户使用Subscribe订阅消息的时候，默认首次offset是latest 其实就是读取左后一个偏移量。 用户可以`auto.offset.reset`控制消费行为
properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // 默认为latest
#自己的笔记
使用AUTO_OFFSET_RESET_CONFIG进行消费行为的时候，首次本组一次是读取0偏移量以后的数据，本组在进行第二次消费后，因为偏移量在第一次已经提交，第二次进行消费的时候就会读取上一次提交偏移量之后的数据。如果换个组，就会重新的从0之后开始读。
因为在kafkaConsumer.poll的时候，kafka的客户端会定时的向kafka服务端提交偏移量，如果用户不想自动提交偏移量，则需要进行手动控制
enable.auto.commit=true
auto.commit.interval.ms=5000
手动偏移量提交
package com.baizhi.test02;
import com.baizhi.test01.ObjectDeserializer;
import com.baizhi.test01.User;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Properties;
public class TestConsumer {
public static void main(String[] args) {
Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092");
/*
* key的反序列化处理
* */
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
/*
* value的反序列化处理
* */
properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ObjectDeserializer.class);
/*
* 指定组名
* */
properties.put(ConsumerConfig.GROUP_ID_CONFIG, "G1");
properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // 默认为latest
properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
/*
* 创建KafkaConsume 对象 订阅某个topic
* */
KafkaConsumer<String, User> kafkaConsumer = new KafkaConsumer<>(properties);
/*
* 订阅topic01
* */
kafkaConsumer.subscribe(Arrays.asList("topic03"));
/*
* 通过kafkaConsumer.poll 获取记录
* */
while (true) {
ConsumerRecords<String, User> consumerRecords = kafkaConsumer.poll(1000);
if (consumerRecords != null && !consumerRecords.isEmpty()) {
HashMap<TopicPartition, OffsetAndMetadata> map = new HashMap<>();
for (ConsumerRecord<String, User> record : consumerRecords) {
String key = record.key();
User user = record.value();
long offset = record.offset();
int partition = record.partition();
long timestamp = record.timestamp();
System.out.println("key:" + key + " value:" + user + " 偏移量:" + offset + " 分区：" + partition + " 时间戳：" + timestamp);
TopicPartition topicPartition = new TopicPartition("topic03", partition);
OffsetAndMetadata metadata = new OffsetAndMetadata(offset + 1);
map.put(topicPartition, metadata);
}
kafkaConsumer.commitSync(map);
}
}
}
}
指定消费者分区
通过assign的方式进行指定分区的消费，那么组的管理策略就失效了。
package com.baizhi.test03;
import com.baizhi.test01.ObjectDeserializer;
import com.baizhi.test01.User;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Properties;
public class TestConsumer {
public static void main(String[] args) {
Properties properties = new Properties();
properties.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "HadoopNode01:9092,HadoopNode02:9092,HadoopNode03:9092");
/*
* key的反序列化处理
* */
properties.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
/*
* value的反序列化处理
* */
properties.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ObjectDeserializer.class);
/*
* 指定组名
* */
properties.put(ConsumerConfig.GROUP_ID_CONFIG, "G1");
properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // 默认为latest
/*
* 创建KafkaConsume 对象 订阅某个topic
* */
KafkaConsumer<String, User> kafkaConsumer = new KafkaConsumer<>(properties);
/*
* 订阅topic01
* */
//kafkaConsumer.subscribe(Arrays.asList("topic04"));
/*
* 指定分区
* */
kafkaConsumer.assign(Arrays.asList(new TopicPartition("topic04", 0)));
kafkaConsumer.seek(new TopicPartition("topic04", 0), 0);
/*
* 通过kafkaConsumer.poll 获取记录
* */
while (true) {
ConsumerRecords<String, User> consumerRecords = kafkaConsumer.poll(1000);
for (ConsumerRecord<String, User> record : consumerRecords) {
String key = record.key();
User user = record.value();
long offset = record.offset();
int partition = record.partition();
long timestamp = record.timestamp();
System.out.println("key:" + key + " value:" + user + " 偏移量:" + offset + " 分区：" + partition + " 时间戳：" + timestamp);
}
}
}
}
通过assign 方式进行消费，手动提交偏移量将不再有意义，因为在seek中指定了可以从哪里开始消费，此时就不用纠结偏移量的概念了，因为偏移量此时有用户决定。
生产者幂等性
https://www.cnblogs.com/javalyy/p/8882144.html 幂等性概念参考
在真正的生产环境中，往往会因为网络等种种原因，导致生产者会重复生产消息。生产者在重试的时候，往往可能前一个消息已经生产过了，但是仍然生了重复的消息。这样就会导致数据紊乱，比如在使用MongoDB的时候，如果不加控制的话有可能会出现了两个重复的用户。所以为了应对此等场景，kafka也有其解决办法，那就是kafka的幂等性
幂等:多次操作最终的影响等价与一次操作称为幂等性操作,所有的读操作一定是幂等的.所有的写操作一定不是幂等的.当 生产者和broker默认有acks应答机制,如果当生产者发送完数据给broker之后如果没有在规定的时间内收到应答,生产者可以考虑重发数据.可以通过一下配置参数提升生产者的可靠性.
acks = all // 0 无需应答 n 应答个数 -1所有都需要
retries = 3 // 表示重试次数
request.timeout.ms = 3000 //等待应答超时时间
enable.idempotence = true //开启幂等性
public class KafkaProducerDemo {
public static void main(String[] args) {
//0.配置生产者了连接属性
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"CentOSA:9092,CentOSB:9092,CentOSC:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.ACKS_CONFIG,"all");//等待所有从机应答
props.put(ProducerConfig.RETRIES_CONFIG,3);//重试3次
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG,3000);//等待3s应答
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG,true);//开启幂等性
//1.创建Kafka生产者
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
//2.构建ProducerRecord
for (int i=15;i<20;i++){
DecimalFormat decimalFormat = new DecimalFormat("000");
User user = new User(i, "name" + i, i % 2 == 0);
ProducerRecord<String, String> record = new ProducerRecord<String, String>("topic06", decimalFormat.format(i), "user"+i);
//3.发送消息
producer.send(record);
}
//4.清空缓冲区
producer.flush();
//5.关闭生产者
producer.close();
}
生产者批量发送
生产者会尝试缓冲record，实现批量发送，通过一下配置控制发送时机，记住如果开启可batch，一定在关闭producer之前需要flush。
batch.size = 16384 //16KB 缓冲16kb数据本地
linger.ms = 2000 //默认逗留时间
public static void main(String[] args) {
//0.配置生产者了连接属性
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"CentOSA:9092,CentOSB:9092,CentOSC:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.ACKS_CONFIG,"all");
props.put(ProducerConfig.RETRIES_CONFIG,3);
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG,3000);
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG,true);
props.put(ProducerConfig.BATCH_SIZE_CONFIG,1024);//1kb缓冲区
props.put(ProducerConfig.LINGER_MS_CONFIG,1000);//设置逗留时常
//1.创建Kafka生产者
KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);
//2.构建ProducerRecord
for (int i=15;i<20;i++){
DecimalFormat decimalFormat = new DecimalFormat("000");
User user = new User(i, "name" + i, i % 2 == 0);
ProducerRecord<String, String> record = new ProducerRecord<String, String>("topic06", decimalFormat.format(i), "user"+i);
//3.发送消息
producer.send(record);
}
//4.清空缓冲区
producer.flush();
//5.关闭生产者
producer.close();
}
生产者事务
kafka生产者事务指的是在发送多个数据的时候，保证多个Record记录发送的原子性。如果有一条发送失败就回退，但是需要注意在使用kafka事务的时候需要调整消费者的事务隔离级别设置为read_committed因为kafka默认的事务隔离策略是read_uncommitted
开启事务
transactional.id=transaction-1 //必须保证唯一
enable.idempotence=true //开启kafka的幂等性
生产者 Only
public class KafkaProducerDemo {
public static void main(String[] args) {
//1.创建Kafka生产者
KafkaProducer<String, String> producer = buildKafkaProducer();
//2.初始化事务和开启事务
producer.initTransactions();
producer.beginTransaction();
try {
for (int i=5;i<10;i++){
DecimalFormat decimalFormat = new DecimalFormat("000");
User user = new User(i, "name" + i, i % 2 == 0);
ProducerRecord<String, String> record = new ProducerRecord<String, String>("topic07", decimalFormat.format(i), "user"+i);
producer.send(record);
}
producer.flush();
//3.提交事务]
producer.commitTransaction();
} catch (Exception e) {
System.err.println(e.getMessage());
//终止事务
producer.abortTransaction();
}
//5.关闭生产者
producer.close();
}
private static KafkaProducer<String, String> buildKafkaProducer() {
//0.配置生产者了连接属性
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"CentOSA:9092,CentOSB:9092,CentOSC:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.ACKS_CONFIG,"all");
props.put(ProducerConfig.RETRIES_CONFIG,3);
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG,3000);
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG,true);
props.put(ProducerConfig.BATCH_SIZE_CONFIG,1024);//1kb缓冲区
props.put(ProducerConfig.LINGER_MS_CONFIG,1000);//设置逗留时常
//开启事务
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG,"transaction-"+UUID.randomUUID().toString());
return new KafkaProducer<String, String>(props);
}
}
消费者那方需要将事务隔离级别设置为`read_committed`
public class KafkaConsumerDemo {
public static void main(String[] args) {
//1.创建Kafka消费者
KafkaConsumer<String, String> consumer = buildKafkaConsumer();
//2.订阅topics
consumer.subscribe(Arrays.asList("topic07"));
//3.死循环读取消息
while(true){
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
if(records!=null && !records.isEmpty()){
for (ConsumerRecord<String, String> record : records) {
int partition = record.partition();
long offset = record.offset();
long timestamp = record.timestamp();
String key = record.key();
String value = record.value();
System.out.println(partition+"\t"+offset+"\t"+timestamp+"\t"+key+"\t"+value);
}
}
}
}
private static KafkaConsumer<String, String> buildKafkaConsumer() {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,"CentOSA:9092,CentOSB:9092,CentOSC:9092");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.GROUP_ID_CONFIG,"group1");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG,"read_committed");
return new KafkaConsumer<String, String>(props);
}
}
生产者&消费者
package com.baizhi.demo08;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.common.serialization.Deserializer;
import org.apache.kafka.common.serialization.Serializer;
import java.util.Properties;
import java.util.UUID;
public class KafkaUtils {
public static KafkaConsumer<String, String> buildKafkaConsumer(String servers, Class<? extends Deserializer> keyDeserializer,
Class<? extends Deserializer> valueDeserializer,String group) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,servers);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,keyDeserializer);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,valueDeserializer);
props.put(ConsumerConfig.GROUP_ID_CONFIG,group);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG,"earliest");
props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG,"read_committed");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,false);//设置为手动提交
return new KafkaConsumer<String, String>(props);
}
public static KafkaProducer<String, String> buildKafkaProducer(String servers, Class<? extends Serializer> keySerializer,
Class<? extends Serializer> valueSerializer) {
//0.配置生产者了连接属性
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,servers);
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,keySerializer);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,valueSerializer);
props.put(ProducerConfig.ACKS_CONFIG,"all");
props.put(ProducerConfig.RETRIES_CONFIG,3);
props.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG,3000);
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG,true);
props.put(ProducerConfig.BATCH_SIZE_CONFIG,1024);//1kb缓冲区
props.put(ProducerConfig.LINGER_MS_CONFIG,1000);//设置逗留时常
//开启事务
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG,"transaction-"+ UUID.randomUUID().toString());
return new KafkaProducer<String, String>(props);
}
}
KafkaProducerAndConsumer
package com.baizhi.demo08;
import com.baizhi.demo05.User;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;
import java.text.DecimalFormat;
import java.time.Duration;
import java.util.*;
public class KafkaProducerAndConsumer {
public static void main(String[] args) {
String servers = "CentOSA:9092,CentOSB:9092,CentOSC:9092";
String group="g1";
//1.创建Kafka生产者
KafkaProducer<String, String> producer = KafkaUtils.buildKafkaProducer(servers, StringSerializer.class, StringSerializer.class);
KafkaConsumer<String, String> consumer = KafkaUtils.buildKafkaConsumer(servers, StringDeserializer.class, StringDeserializer.class,group);
consumer.subscribe(Arrays.asList("topic08"));
//初始化事务
producer.initTransactions();
while (true) {
producer.beginTransaction();
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
try {
Map<TopicPartition, OffsetAndMetadata> commits = new HashMap<TopicPartition, OffsetAndMetadata>();
for (ConsumerRecord<String, String> record : records) {
TopicPartition partition = new TopicPartition(record.topic(), record.partition());
OffsetAndMetadata offsetAndMetadata = new OffsetAndMetadata(record.offset() + 1);
commits.put(partition, offsetAndMetadata);
System.out.println(record);
ProducerRecord<String, String> srecord = new ProducerRecord<String, String>("topic09", record.key(), record.value());
producer.send(srecord);
}
producer.flush();
//并没使用 consumer提交，而是使用producer帮助消费者提交偏移量
producer.sendOffsetsToTransaction(commits,group);
//提交生产者的偏移量
producer.commitTransaction();
} catch (Exception e) {
//System.err.println(e.getMessage());
producer.abortTransaction();
}
}
}
}
七、集成SpringBoot
依赖
pom.xml
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.baizhi</groupId>
<artifactId>Kafka_TestSpringBoot</artifactId>
<packaging>war</packaging>
<version>1.0-SNAPSHOT</version>
<name>KafkaSpringBoot Maven Webapp</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<java.version>1.8</java.version>
<kafka.version>2.2.0</kafka.version>
</properties>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.5.RELEASE</version>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>2.2.5.RELEASE</version>
</dependency>
<!-- kafka client处理 -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka.version}</version>
</dependency>
</dependencies>
<build>
<finalName>KafkaSpringBoot</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<dependencies>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>springloaded</artifactId>
<version>1.2.7.RELEASE</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>
配置文件
- application.properties
server.port=8888
# 生产者
spring.kafka.producer.bootstrap-servers=CentOSA:9092,CentOSB:9092,CentOSC:9092
spring.kafka.producer.acks=all
spring.kafka.producer.retries=1
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
# 消费者
spring.kafka.consumer.bootstrap-servers=CentOSA:9092,CentOSB:9092,CentOSC:9092
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
- application.yml
spring:
kafka:
consumer:
bootstrap-servers: CentOSA:9092,CentOSB:9092,CentOSC:9092
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
producer:
bootstrap-servers: CentOSA:9092,CentOSB:9092,CentOSC:9092
acks: all
retries: 1
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
server:
port: 8888
操作
- KafkaApplicationDemo
@SpringBootApplication
@EnableScheduling
public class KafkaApplicationDemo {
@Autowired
private KafkaTemplate kafkaTemplate;
public static void main(String[] args) {
SpringApplication.run(KafkaApplicationDemo.class,args);
}
@Scheduled(cron = "0/1 * * * * ?")
public void send(){
String[] message=new String[]{"this is a demo","hello world","hello boy"};
ListenableFuture future = kafkaTemplate.send("topic07", message[new Random().nextInt(message.length)]);
future.addCallback(o -> System.out.println("send-消息发送成功：" + message), throwable -> System.out.println("消息发送失败：" + message));
}
@KafkaListener(topics = "topic07",id="g1")
public void processMessage(ConsumerRecord<?, ?> record) {
System.out.println("record:"+record);
}
}
endency>
</plugins>
## 配置文件
- application.properties
```properties
server.port=8888
# 生产者
spring.kafka.producer.bootstrap-servers=CentOSA:9092,CentOSB:9092,CentOSC:9092
spring.kafka.producer.acks=all
spring.kafka.producer.retries=1
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer
# 消费者
spring.kafka.consumer.bootstrap-servers=CentOSA:9092,CentOSB:9092,CentOSC:9092
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer
- application.yml
spring:
kafka:
consumer:
bootstrap-servers: CentOSA:9092,CentOSB:9092,CentOSC:9092
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
producer:
bootstrap-servers: CentOSA:9092,CentOSB:9092,CentOSC:9092
acks: all
retries: 1
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
server:
port: 8888
操作
- KafkaApplicationDemo
@SpringBootApplication
@EnableScheduling
public class KafkaApplicationDemo {
@Autowired
private KafkaTemplate kafkaTemplate;
public static void main(String[] args) {
SpringApplication.run(KafkaApplicationDemo.class,args);
}
@Scheduled(cron = "0/1 * * * * ?")
public void send(){
String[] message=new String[]{"this is a demo","hello world","hello boy"};
ListenableFuture future = kafkaTemplate.send("topic07", message[new Random().nextInt(message.length)]);
future.addCallback(o -> System.out.println("send-消息发送成功：" + message), throwable -> System.out.println("消息发送失败：" + message));
}
@KafkaListener(topics = "topic07",id="g1")
public void processMessage(ConsumerRecord<?, ?> record) {
System.out.println("record:"+record);
}
}
224
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
