Kafka1.0日常维护

最新推荐文章于 2023-11-13 15:34:02 发布
置顶 最新推荐文章于 2023-11-13 15:34:02 发布
阅读量981 收藏 3
点赞数
分类专栏： Kafka 文章标签： 大数据 bigdata kafka
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/gpf19950411/article/details/79430237
版权

日常维护操作

4.1 创建topic两个备份6个分区：

./kafka-topics.sh --create --zookeeper app212:2181 --replication-factor 2 --partitions 6 --topic t_test

4.2 查看存在的topic：

./kafka-topics.sh --zookeeper app212:2181 --list

4.3 查看topic详细信息：


./kafka-topics.sh --describe --zookeeper app212:2181 --topic t_test

topic详细信息

4.4 生产数据：

./kafka-console-producer.sh --broker-list localhost:9092 --topic t_test

模拟生产数据

4.5 消费数据：

消费数据：
./kafka-console-consumer.sh --zookeeper app211:2181 --topic t_test

模拟消费数据

4.6 查看topic某分区偏移量最大（小）值

./kafka-run-class.sh kafka.tools.GetOffsetShell --topic t_test  --time -1 --broker-list 10.242.167.216:9092,10.242.167.217:9092,10.242.167.218:9092 --partitions 0
time为-1时表示最大值，time为-2时表示最小值

偏移量

4.7 删除topic

只会删除zookeeper中的元数据，消息文件须手动删除

./kafka-topics.sh --zookeeper app212:2181 --delete --topic t_test

4.8 增加topic分区数（只能增加）

./kafka-topics.sh --zookeeper app212:2181  --alter --topic t_test --partitions 10

4.8.1 查看group消费情况

kafka-consumer-groups.sh --bootstrap-server host:9092 --group group_name --describe

4.9 增加topic副本数

创建partitions-extension-push-token-topic.json
vim /opt/kafka/partitions-extension-push-token-topic.json

{
    "partitions": [
        {
            "topic": "t_test",
            "partition": 0,
            "replicas": [
                4,
                5,
				6
            ]
        },
        {
            "topic": "t_test",
            "partition": 1,
            "replicas": [
                5,
                6,
				1
            ]
        },
        {
            "topic": "t_test",
            "partition": 2,
            "replicas": [
                6,
                1,
				2
            ]
        },
		{
            "topic": "t_test",
            "partition": 3,
            "replicas": [
                1,
                2,
				3
            ]
        },
		{
            "topic": "t_test",
            "partition": 4,
            "replicas": [
                2,
                3,
				4
            ]
        },
		{
            "topic": "t_test",
            "partition": 5,
            "replicas": [
                3,
                4,
				5
            ]
        }
    ],
    "version": 1
}

执行变更：

命令：./kafka-reassign-partitions.sh --zookeeper localhost:2181  --reassignment-json-file

[root@dbmspreapp212 bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181  --reassignment-json-file  ../partitions-extension-push-token-topic.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"t_test","partition":1,"replicas":[5,6],"log_dirs":["any","any"]},{"topic":"t_test","partition":5,"replicas":[3,4],"log_dirs":["any","any"]},{"topic":"t_test","partition":2,"replicas":[6,1],"log_dirs":["any","any"]},{"topic":"t_test","partition":3,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"t_test","partition":4,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"t_test","partition":0,"replicas":[4,5],"log_dirs":["any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

验证结果：

./kafka-reassign-partitions.sh --zookeeper localhost:2181  --reassignment-json-file  ../partitions-extension-push-token-topic.json --verify

[root@dbmspreapp212 bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181  --reassignment-json-file  ../partitions-extension-push-token-topic.json --verify
Status of partition reassignment: 
Reassignment of partition t_test-5 completed successfully
Reassignment of partition t_test-2 completed successfully
Reassignment of partition t_test-0 completed successfully
Reassignment of partition t_test-4 completed successfully
Reassignment of partition t_test-1 completed successfully
Reassignment of partition t_test-3 completed successfully

查看结果
变更前：

[root@dbmspreapp212 bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic t_test
Topic:t_test	PartitionCount:6	ReplicationFactor:2	Configs:
	Topic: t_test	Partition: 0	Leader: 4	Replicas: 4,5	Isr: 4,5
	Topic: t_test	Partition: 1	Leader: 5	Replicas: 5,6	Isr: 5,6
	Topic: t_test	Partition: 2	Leader: 6	Replicas: 6,1	Isr: 6,1
	Topic: t_test	Partition: 3	Leader: 1	Replicas: 1,2	Isr: 1,2
	Topic: t_test	Partition: 4	Leader: 2	Replicas: 2,3	Isr: 2,3
	Topic: t_test	Partition: 5	Leader: 3	Replicas: 3,4	Isr: 3,4

变更后

[root@dbmspreapp212 bin]# ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic t_test
Topic:t_test	PartitionCount:6	ReplicationFactor:3	Configs:
	Topic: t_test	Partition: 0	Leader: 4	Replicas: 4,5,6	Isr: 4,5,6
	Topic: t_test	Partition: 1	Leader: 5	Replicas: 5,6,1	Isr: 5,6,1
	Topic: t_test	Partition: 2	Leader: 6	Replicas: 6,1,2	Isr: 6,1,2
	Topic: t_test	Partition: 3	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
	Topic: t_test	Partition: 4	Leader: 2	Replicas: 2,3,4	Isr: 2,3,4
	Topic: t_test	Partition: 5	Leader: 3	Replicas: 3,4,5	Isr: 3,4,5

4.10 减少topic副本数

类似于增加副本数，只需要更改json文件的replicas配置项

4.11 Kafka集群扩容（重分配topic分区）

创建topic-to-move.json
vim /opt/kafka/topic-to-move.json

{
    "topics": [
        {
            "topic": "t_test"
        }
    ],
    "version": 1
}

生成迁移分配规则json文件:

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file ../topic-to-move.json --broker-list "1,2,3,4,5,6,8" --generate

[root@dbmspreapp212 bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file ../topic-to-move.json --broker-list "1,2,3,4,5,6,8" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"t_test","partition":1,"replicas":[6,1],"log_dirs":["any","any"]},{"topic":"t_test","partition":5,"replicas":[4,5],"log_dirs":["any","any"]},{"topic":"t_test","partition":6,"replicas":[5,6],"log_dirs":["any","any"]},{"topic":"t_test","partition":2,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"t_test","partition":3,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"t_test","partition":4,"replicas":[3,4],"log_dirs":["any","any"]},{"topic":"t_test","partition":0,"replicas":[5,6],"log_dirs":["any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"t_test","partition":1,"replicas":[1,8],"log_dirs":["any","any"]},{"topic":"t_test","partition":5,"replicas":[5,4],"log_dirs":["any","any"]},{"topic":"t_test","partition":6,"replicas":[6,5],"log_dirs":["any","any"]},{"topic":"t_test","partition":2,"replicas":[2,1],"log_dirs":["any","any"]},{"topic":"t_test","partition":3,"replicas":[3,2],"log_dirs":["any","any"]},{"topic":"t_test","partition":4,"replicas":[4,3],"log_dirs":["any","any"]},{"topic":"t_test","partition":0,"replicas":[8,6],"log_dirs":["any","any"]}]}

其中的Current partition replica assignment指的是迁移前的partition replica；Proposed partition reassignment configuration 指的就是迁移分配规则json。需要将该json文件保存到json文件中(如expand-cluster-reassignment.json)
vim /opt/kafka/expand-cluster-reassignment.json

{"version":1,"partitions":[{"topic":"t_test","partition":1,"replicas":[1,8],"log_dirs":["any","any"]},{"topic":"t_test","partition":5,"replicas":[5,4],"log_dirs":["any","any"]},{"topic":"t_test","partition":6,"replicas":[6,5],"log_dirs":["any","any"]},{"topic":"t_test","partition":2,"replicas":[2,1],"log_dirs":["any","any"]},{"topic":"t_test","partition":3,"replicas":[3,2],"log_dirs":["any","any"]},{"topic":"t_test","partition":4,"replicas":[4,3],"log_dirs":["any","any"]},{"topic":"t_test","partition":0,"replicas":[8,6],"log_dirs":["any","any"]}]}

执行迁移分配

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file ../expand-cluster-reassignment.json --execute

[root@dbmspreapp212 bin]# kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file ../expand-cluster-reassignment.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"t_test","partition":1,"replicas":[6,1],"log_dirs":["any","any"]},{"topic":"t_test","partition":5,"replicas":[4,5],"log_dirs":["any","any"]},{"topic":"t_test","partition":6,"replicas":[5,6],"log_dirs":["any","any"]},{"topic":"t_test","partition":2,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"t_test","partition":3,"replicas":[2,3],"log_dirs":["any","any"]},{"topic":"t_test","partition":4,"replicas":[3,4],"log_dirs":["any","any"]},{"topic":"t_test","partition":0,"replicas":[5,6],"log_dirs":["any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

验证

./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file ../expand-cluster-reassignment.json --verify

[root@dbmspreapp212 bin]# ./kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file ../expand-cluster-reassignment.json --verify
Status of partition reassignment: 
Reassignment of partition t_test-5 completed successfully
Reassignment of partition t_test-2 completed successfully
Reassignment of partition t_test-0 completed successfully
Reassignment of partition t_test-4 completed successfully
Reassignment of partition t_test-6 completed successfully
Reassignment of partition t_test-1 completed successfully
Reassignment of partition t_test-3 completed successfully

结果：

[root@dbmspreapp212 bin]# ./kafka-topics.sh --zookeeper localhost:2181 --describe --topic t_test
Topic:t_test	PartitionCount:7	ReplicationFactor:2	Configs:
	Topic: t_test	Partition: 0	Leader: 8	Replicas: 8,6	Isr: 6,8
	Topic: t_test	Partition: 1	Leader: 1	Replicas: 1,8	Isr: 1,8
	Topic: t_test	Partition: 2	Leader: 1	Replicas: 2,1	Isr: 1,2
	Topic: t_test	Partition: 3	Leader: 2	Replicas: 3,2	Isr: 2,3
	Topic: t_test	Partition: 4	Leader: 3	Replicas: 4,3	Isr: 3,4
	Topic: t_test	Partition: 5	Leader: 4	Replicas: 5,4	Isr: 4,5
	Topic: t_test	Partition: 6	Leader: 5	Replicas: 6,5	Isr: 5,6

4.12 Kafka集群节点下线

和扩容相反，--broker-list "1,2,3,4,5,8",比如下线节点6，然后生成文件等进行操作。
重新leader、重新均衡,下线节点完成后重新均衡下
./kafka-preferred-replica-election.sh --zookeeper localhost:2181

GPF19950411
关注
专栏目录
kafka集群部署文档(部署，运维，FAQ)
06-27
kafka集群部署文档及kafka详解，包括kafka常用命令，详细部署说明，如何运维以及一些FAQ，下载app注册免费获取：http://m3w.cn/jcsh
kafka日常运维工作
weixin_43046724的博客
02-26 1903
一、创建/删除/重建topic 1、创建：./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 10 --topicTOPIC_NAME 2、删除：./bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topicTOPIC_NAME 3、查看状态和分区负载详情：./bin/kafka-topics.s..
kafka维护工具使用指南
热门推荐
xhpscdx的专栏
08-04 1万+
一.前言 kafka-manager是Yahoo开源的Kafka管理工具，并在持续提交。是目前最好用kafka监控开源软件。 由于文档较少，scala开发，对于有些术语的确无法理解，请抱歉。 本文也介绍运维常用kafka命令。 文档根据大家反馈，会持续更新，请关注. 二.kafka-manager介绍 管理多个集群 方便查看集群状态 执行preferred replica election 批量为多
Kafka运维操作指南
zhinengyunwei的博客
01-14 763
作者：石文 时间：2018-12-14 1.启停，进程状态 #启动。若要开启JMX端口需要加载环境变量 su hadoop ./bin/kafka-server-start.sh -daemon ./config/server.properties #停止。 su hadoop ./bin/kafka-server-stop.sh 2.topic相关 #创建topic.默认数据保留时间为7天，...
kafka运维命令大全
别来沾边儿
07-06 394
相关可选参数支持正则表达式匹配Topic来进行删除,只需要将topic 用双引号包裹起来 例如: 删除以为开头的topic;删除任意Topic (慎用)zk方式(不推荐)kafka版本 >= 2.2 支持下面方式（推荐）单个Topic扩容批量扩容 (将所有正则表达式匹配到的Topic分区扩容到4个) 正则表达式的意思是匹配所有; 您可按需匹配PS: 当某个Topic的分区少于指定的分区数时候,他会抛出异常;但是不会影响其他Topic正常进行;相关可选参数 | 参数 |描述 |例子| |--|--|--| |
kafka处理超大消息的配置 org.apache.kafka.common.errors.RecordTooLargeException
01-07
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server ...
kafka0.10.2.1集群搭建.docx
06-11
- 从 Apache Kafka 官网（http://kafka.apache.org/downloads）下载相应版本，解压到指定目录。 3. **配置环境变量**： - 编辑 `/etc/profile` 文件，添加 Kafka 的路径，并通过 `source /etc/profile` 使其生效...
kafka学习kafka学习.zip
最新发布
02-05
1. **日志收集**：Kafka常用于收集各种系统的日志数据，提供统一的数据入口。 2. **实时处理**：与Storm或Spark结合，实现实时数据流处理。 3. **消息系统**：替代传统的消息队列，如RabbitMQ，提供更高效的性能。...
Apache Kafka实战.pdf
02-21
Kafka是一个高吞吐量、低延迟的消息发布订阅系统，常用于构建实时数据管道和流应用程序。以下是根据书中的内容提取的关键知识点： 1. **Kafka基础**：Kafka是一个分布式流处理平台，它允许发布和订阅持久化的消息流...
springboot集成kafka0.10.0.1并发送邮件项目
07-11
spring.kafka.bootstrap-servers=localhost:9092 ``` 接着，我们可以创建一个Kafka消费者来监听特定的topic。Spring Boot提供了`@KafkaListener`注解，可以用于标记监听方法。这个方法会接收到从Kafka topic推送的...
Kafka常见运维命令.md
05-31
Kafka常见运维命令文档，Kafka常见运维命令文档，Kafka常见运维命令文档
kafka使用手册
02-01
kafka入门与使用的详细教程，你值得拥有。
Kafka_2.11-1.1.0
05-06
Kafka_2.11-1.1.0
一篇总结Kafka系统运维及问题解析的文章
love6a6的博客
11-13 427
Kafka是一个全方位的流处理平台，用于构建实时数据流应用程序和管道数据。在Kafka系统的运维中，有几个关键方面需要考虑，包括架构设计、部署、监控、容错性和性能优化。
15、kafka集群维护
lixiang987654321的专栏
08-16 338
kafka集群维护1、集群信息实时查看工具（topic工具）2、kafka集群leader平衡机制3、集群分区日志迁移 1、集群信息实时查看工具（topic工具） （1）列出集群当前所有可用的topic kafka-topics.sh --list --zookeeper kafka1:2181 （2）查看集群特定的topic信息 /bin/kafka-topics.sh --describe ...
kafka维护问题总结
weixin_33895657的博客
12-08 328
2019独角兽企业重金招聘Python工程师标准>>> ...
Kafka运维大全来了！优化、监控、故障处理……
过往记忆大数据
10-24 1839
作者介绍小火牛，项目管理高级工程师，具有多年大数据平台运维管理及开发优化经验。管理过多个上千节点集群，擅长对外多租户平台的维护开发。信科院大数据性能测试、功能测试主力，大...
Kafka面试V1.0
weixin_43994052的博客
10-19 355
Kafka面试一篇搞定
kafka 运维中遇到的问题
h952520296的博客
12-08 1万+
1，java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiledkafka 进程失败，重启失败 检查日志发现这个，原因是磁盘满了，清除 kafka-logs 即可2，kafka定时清理日志不生效网上很多文档，说是要设置log.retention.hour等等参数。 默认是保留7天，但我实测下来发现日志根本没有任何变化。原因是因为kafka只会回收上个分片的数据配置没有生效的原因
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值