kafka笔记
一.zookeeper
- 启动zookeeper
启动服务：zkServer.sh start
zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
服务已开启
查看状态：zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: standalone
这里显示服务是单实例的
关闭服务：zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /usr/zookeeper/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
服务已停止
- 创建节点
启用客户端：./zkCli.sh
Connecting to localhost:2181
2021-03-06 00:59:21,242 [myid:] - INFO [main:Environment@98] - Client environment:zookeeper.version=3.6.2--803c7f1a12f85978cb049af5e4ef23bd8b688715, built on 09/04/2020 12:44 GMT
2021-03-06 00:59:21,249 [myid:] - INFO [main:Environment@98] - Client environment:host.name=localhost
2021-03-06 00:59:21,249 [myid:] - INFO [main:Environment@98] - Client environment:java.version=1.8.0_281
2021-03-06 00:59:21,254 [myid:] - INFO [main:Environment@98] - Client environment:java.vendor=Oracle Corporation
2021-03-06 00:59:21,254 [myid:] - INFO [main:Environment@98] - Client environment:java.home=/usr/jdk1.8.0/jre
2021-03-06 00:59:21,254 [myid:] - INFO [main:Environment@98] - Client environment:java.class.path=/usr/zookeeper/bin/../zookeeper-server/target/classes:/usr/zookeeper/bin/../build/classes:/usr/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/usr/zookeeper/bin/../build/lib/*.jar:/usr/zookeeper/bin/../lib/zookeeper-prometheus-metrics-3.6.2.jar:/usr/zookeeper/bin/../lib/zookeeper-jute-3.6.2.jar:/usr/zookeeper/bin/../lib/zookeeper-3.6.2.jar:/usr/zookeeper/bin/../lib/snappy-java-1.1.7.jar:/usr/zookeeper/bin/../lib/slf4j-log4j12-1.7.25.jar:/usr/zookeeper/bin/../lib/slf4j-api-1.7.25.jar:/usr/zookeeper/bin/../lib/simpleclient_servlet-0.6.0.jar:/usr/zookeeper/bin/../lib/simpleclient_hotspot-0.6.0.jar:/usr/zookeeper/bin/../lib/simpleclient_common-0.6.0.jar:/usr/zookeeper/bin/../lib/simpleclient-0.6.0.jar:/usr/zookeeper/bin/../lib/netty-transport-native-unix-common-4.1.50.Final.jar:/usr/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.50.Final.jar:/usr/zookeeper/bin/../lib/netty-transport-4.1.50.Final.jar:/usr/zookeeper/bin/../lib/netty-resolver-4.1.50.Final.jar:/usr/zookeeper/bin/../lib/netty-handler-4.1.50.Final.jar:/usr/zookeeper/bin/../lib/netty-common-4.1.50.Final.jar:/usr/zookeeper/bin/../lib/netty-codec-4.1.50.Final.jar:/usr/zookeeper/bin/../lib/netty-buffer-4.1.50.Final.jar:/usr/zookeeper/bin/../lib/metrics-core-3.2.5.jar:/usr/zookeeper/bin/../lib/log4j-1.2.17.jar:/usr/zookeeper/bin/../lib/json-simple-1.1.1.jar:/usr/zookeeper/bin/../lib/jline-2.14.6.jar:/usr/zookeeper/bin/../lib/jetty-util-9.4.24.v20191120.jar:/usr/zookeeper/bin/../lib/jetty-servlet-9.4.24.v20191120.jar:/usr/zookeeper/bin/../lib/jetty-server-9.4.24.v20191120.jar:/usr/zookeeper/bin/../lib/jetty-security-9.4.24.v20191120.jar:/usr/zookeeper/bin/../lib/jetty-io-9.4.24.v20191120.jar:/usr/zookeeper/bin/../lib/jetty-http-9.4.24.v20191120.jar:/usr/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/usr/zookeeper/bin/../lib/jackson-databind-2.10.3.jar:/usr/zookeeper/bin/../lib/jackson-core-2.10.3.jar:/usr/zookeeper/bin/../lib/jackson-annotations-2.10.3.jar:/usr/zookeeper/bin/../lib/commons-lang-2.6.jar:/usr/zookeeper/bin/../lib/commons-cli-1.2.jar:/usr/zookeeper/bin/../lib/audience-annotations-0.5.0.jar:/usr/zookeeper/bin/../zookeeper-*.jar:/usr/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/usr/zookeeper/bin/../conf:.:/usr/jdk1.8.0/lib:/usr/jdk1.8.0/jre/lib
2021-03-06 00:59:21,255 [myid:] - INFO [main:Environment@98] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2021-03-06 00:59:21,256 [myid:] - INFO [main:Environment@98] - Client environment:java.io.tmpdir=/tmp
2021-03-06 00:59:21,256 [myid:] - INFO [main:Environment@98] - Client environment:java.compiler=<NA>
2021-03-06 00:59:21,256 [myid:] - INFO [main:Environment@98] - Client environment:os.name=Linux
2021-03-06 00:59:21,256 [myid:] - INFO [main:Environment@98] - Client environment:os.arch=amd64
2021-03-06 00:59:21,257 [myid:] - INFO [main:Environment@98] - Client environment:os.version=3.10.0-229.el7.x86_64
2021-03-06 00:59:21,257 [myid:] - INFO [main:Environment@98] - Client environment:user.name=root
2021-03-06 00:59:21,257 [myid:] - INFO [main:Environment@98] - Client environment:user.home=/root
2021-03-06 00:59:21,257 [myid:] - INFO [main:Environment@98] - Client environment:user.dir=/usr/zookeeper/bin
2021-03-06 00:59:21,258 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.free=23MB
2021-03-06 00:59:21,263 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.max=228MB
2021-03-06 00:59:21,264 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.total=29MB
2021-03-06 00:59:21,276 [myid:] - INFO [main:ZooKeeper@1006] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@379619aa
2021-03-06 00:59:21,296 [myid:] - INFO [main:X509Util@77] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2021-03-06 00:59:21,313 [myid:] - INFO [main:ClientCnxnSocket@239] - jute.maxbuffer value is 1048575 Bytes
2021-03-06 00:59:21,327 [myid:] - INFO [main:ClientCnxn@1716] - zookeeper.request.timeout value is 0. feature enabled=false
Welcome to ZooKeeper!
2021-03-06 00:59:21,344 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1167] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181.
2021-03-06 00:59:21,345 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1169] - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2021-03-06 00:59:21,369 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@999] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:49064, server: localhost/0:0:0:0:0:0:0:1:2181
JLine support is enabled
2021-03-06 00:59:21,433 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1433] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, session id = 0x100003119ef0000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
创建节点：create /test demo
[zk: localhost:2181(CONNECTED) 0] create /test demo
Created /test
查看节点：get /test
[zk: localhost:2181(CONNECTED) 1] get /test
demo
查看根节点信息：ls /
[zk: localhost:2181(CONNECTED) 5] ls /
[admin, brokers, cluster, config, consumers, controller_epoch, feature, isr_change_notification, latest_producer_id_block, log_dir_event_notification, test, zookeeper]
更新信息：set /node message
[zk: localhost:2181(CONNECTED) 13] set /test abc
[zk: localhost:2181(CONNECTED) 14] get /test
abc
[zk: localhost:2181(CONNECTED) 15] set /test demo
[zk: localhost:2181(CONNECTED) 16] get /test
demo
删除节点：delete /node
[zk: localhost:2181(CONNECTED) 27] delete /test/demo
[zk: localhost:2181(CONNECTED) 28] delete /test
退出客户端：quit
[zk: localhost:2181(CONNECTED) 30] quit
WATCHER::
WatchedEvent state:Closed type:None path:null
2021-03-06 01:17:57,035 [myid:] - INFO [main:ZooKeeper@1619] - Session: 0x100003119ef0000 closed
2021-03-06 01:17:57,035 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@577] - EventThread shut down for session: 0x100003119ef0000
2021-03-06 01:17:57,053 [myid:] - ERROR [main:ServiceUtils@42] - Exiting JVM with code 0
[root@localhost bin]#
二.kafka
- 启动zookeeper
关闭服务：zkServer.sh stop
- 启动kafka
开启：bin/kafka-server-start.sh config/server.properties
- 测试创建topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic demo
Error while executing topic command : Topic 'demo' already exists.
[2021-03-06 01:45:13,490] ERROR org.apache.kafka.common.errors.TopicExistsException: Topic 'demo' already exists.
(kafka.admin.TopicCommand$)
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic demo2
Created topic demo2.
- 开启一个producer线程
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic demo
[root@localhost kafka]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic demo
>see you again
>
- 开启一个consumer线程
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo --from-beginning
[root@localhost kafka]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo --from-beginning
see you again
- 查看topic
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic demo
[root@localhost kafka]# bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic demo
Topic: demo PartitionCount: 1 ReplicationFactor: 1 Configs:
Topic: demo Partition: 0 Leader: 0 Replicas: 0 Isr: 0
[root@localhost kafka]# bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic demo2
Topic demo2 is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
三.kafka配置(version2.7.0)
- broker配置
server.properties
############################# Server Basics #############################
# 部署kafka实例的标识，标识集群中的broker,该id唯一，且必须是整数
broker.id=0
############################# Socket Server Settings #############################
# broker监听端口
# 如果没有配置可以通过java.net.InetAddress.getCanonicalHostName() 来获取.
# FORMAT:
# listeners = listener_name://host_name:port
# EXAMPLE:
# listeners = PLAINTEXT://your.host.name:9092
#listeners=PLAINTEXT://:9092
#点的主机名会通知给生产者和消费者。如果没有设置，如果配置了"listeners"就使用"listeners"的值。
# 否则就使用java.net.InetAddress.getCanonicalHostName()的返回值
#advertised.listeners=PLAINTEXT://your.host.name:9092
# 将监听器名称映射到安全协议，默认情况下它们是相同的
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
# 处理请求的线程数量
num.network.threads=3
# 该broker处理磁盘IO 的线程数，数值应该大于你的硬盘数
num.io.threads=8
# socket server发送数据缓冲区
socket.send.buffer.bytes=102400
# socket server接收数据缓冲区
socket.receive.buffer.bytes=102400
# socket服务所能够接受的最大的请求量，防止出现OOM，默认值为：100m（应该是socker server所能接受的一个请求的最大大小，默认为100M）
socket.request.max.bytes=104857600
############################# Log Basics #############################
# 用逗号分隔的目录列表，用于存储kafka接受到的数据
log.dirs=/tmp/kafka-logs
# 每一个topic所对应的log的partition分区数目，默认1个。partition数目增大会提高消费并行度，但是也会导致在kafka集群中有更多的文件进行传，partition是kafka分布式存储的核心，一个topic产生的数据拆开打在不同的partition中
num.partitions=1
# 每个数据目录用于启动时日志恢复和关闭时刷新的线程数。当数据目录位于RAID组中时，建议增加该值。
num.recovery.threads.per.data.dir=1
############################# Internal Topic Settings #############################
# 组元数据内部主题“__consumer_offsets”和“__transaction_state”的复制因子对于开发测试以外的任何内容，建议使用大于1的值，以确保可用性，例如3
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
############################# Log Flush Policy #############################
# 消息被立即写入文件系统，但是默认情况下我们只使用fsync()进行同步操作系统缓存懒惰。以下配置控制将数据刷新到磁盘。
#这里有几个重要的权衡:
# 1。持久性:如果不使用复制，未刷新的数据可能会丢失。
# 2。延迟:当刷新发生时，非常大的刷新间隔可能会导致延迟峰值，因为有很多数据需要刷新。
# 3。吞吐量:刷新通常是成本最高的操作，较小的刷新间隔可能会导致过多的查找。
#下面的设置允许配置刷新策略以在一段时间或之后刷新数据每N条消息(或同时)。这可以全局执行，并在每个主题的基础上覆盖,这两个选项可以选择配置一个,当然也可以两个都配置，默认情况下两个都配置，配置如下。
#在强制将数据刷新到磁盘之前要接受的消息数量
#log.flush.interval.messages=10000
# 强制刷新之前，一条消息可以在日志中保存的最大时间
#log.flush.interval.ms=1000
############################# Log Retention Policy #############################
#以下配置控制日志段的处理政策.设置在一段时间后删除段。只要满足其中一个策略（基于时间或基于大小),一个段将被删除，删除总是发生从日志的末尾开始。
#删除日志文件的最小存活时间，默认168小时，即一周
log.retention.hours=168
# 一个基于大小的日志保留策略。段将被从日志中删除只要剩下的部分段不低于log.retention.bytes
#log.retention.bytes=1073741824
#每一个日志段大小的最大值。当到达这个大小时，会生成一个新的片段
log.segment.bytes=1073741824
# 检查日志段的时间间隔，看是否可以根据保留策略删除它们
log.retention.check.interval.ms=300000
############################# Zookeeper #############################
# Zookeeper集群链接地址。这是一个逗号分隔的主机:端口对，每个对应一个zk服务器。如。“127.0.0.1:3000 127.0.0.1:3001 127.0.0.1:3002”。您还可以在url中附加一个可选的chroot字符串来指定
zookeeper.connect=localhost:2181
# zookeeper链接超时时间
zookeeper.connection.timeout.ms=18000
############################# Group Coordinator Settings #############################
# 下面的配置指定GroupCoordinator将延迟初始消费者再平衡的时间(以毫秒为单位)。当新成员加入组时，再平衡将被group.initial.rebalance.delay.ms的值进一步延迟，直到max.poll.interval.ms的最大值。
#默认值是3秒。
#我们在这里重写为0，因为它为开发和测试提供了更好的开箱即用体验。
#但是，在生产环境中，默认值3秒更合适，因为这将有助于避免在应用程序启动时进行不必要的、潜在的昂贵的重新平衡。
group.initial.rebalance.delay.ms=0
- consumer配置
consumer.properties
# Kafka群集信息列表，用于连接kafka集群，如果集群中机器数很多，只需要指定部分的机器主机的信息即可，格式: host1:port1,host2:port2 ...
bootstrap.servers=localhost:9092
# 标识一个consumer组的名称
group.id=test-consumer-group
# 消费者读取偏移量无效的情况下，需要重置消费起始位置，默认为latest（从消费者启动后生成的记录），另外一个选项值是 earliest，将从有效的最小位移位置开始消费: latest, earliest, none
#auto.offset.reset=
#消息key反序列化器
key.deserializer
#消息value反序列化器
value.deserializer
#消费者和群组协调器的最大心跳时间，如果超过该时间则认为该消费者已经死亡或者故障，需要提出消费者组
session .timeout.ms
#一次poll间隔最大时间
max.poll.interval .ms
#consumer提交方式设置
enable.auto.commit
#consumer端一次拉取数据的最大字节数
fetch.max.bytes
#consumer端一次拉取数据的最大条数
max.poll.records
- producer配置
producer.properties
############################# Producer Basics #############################
# Kafka群集信息列表，用于连接kafka集群，如果集群中机器数很多，只需要指定部分的机器主机的信息即可，格式: host1:port1,host2:port2
bootstrap.servers=localhost:9092
# 为所有生成的数据指定压缩编解码器:none, gzip, snappy, lz4, zstd
compression.type=none
# name of the partitioner class for partitioning events; default partition spreads data randomly
#partitioner.class=
# the maximum amount of time the client will wait for the response of a request
#request.timeout.ms=
# how long `KafkaProducer.send` and `KafkaProducer.partitionsFor` will block for
#max.block.ms=
# the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together
#linger.ms=
# the maximum size of a request in bytes
#max.request.size=
# the default batch size in bytes when batching multiple records sent to a partition
#batch.size=
# the total bytes of memory the producer can use to buffer records waiting to be sent to the server
#buffer.memory=
#生产者要求领导者在确认请求完成之前已收到的确认数。这控制了发送记录的持久性。允许以下设置：
#acks=0如果设置为零，那么生产者将完全不等待服务器的任何确认。该记录将立即添加到发送缓冲区中并视为已发送。在这种情况下，retries不能保证服务器已收到记录，并且配置将不会生效（因为客户端通常不会知道任何故障）。为每条记录提供的偏移量将始终设置为-1。
#acks=1这意味着领导者会将记录写到其本地日志中，但是会在不等待所有关注者完全确认的情况下做出响应。在这种情况下，如果领导者在确认记录后立即失败，但是在跟随者复制记录之前，则记录将丢失。
#acks=all这意味着领导者将等待全套同步副本确认记录。这保证了只要至少一个同步副本仍处于活动状态，记录就不会丢失。这是最有力的保证。这等效于acks = -1设置。
#acks=
4900
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
